Sep 16 18:45:00.072415 ip-10-0-129-125 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:45:00.499474 ip-10-0-129-125 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:00.499474 ip-10-0-129-125 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:45:00.499474 ip-10-0-129-125 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:00.499474 ip-10-0-129-125 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:45:00.499474 ip-10-0-129-125 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:00.500548 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.500455    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:45:00.504713 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504684    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:00.504713 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504712    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:00.504713 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504717    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:00.504713 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504720    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:00.504713 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504722    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:00.504713 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504725    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504729    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504731    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504734    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504739    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504743    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504746    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504749    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504752    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504755    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504758    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504761    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504763    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504767    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504770    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504772    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504776    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504779    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504781    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504784    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:00.504890 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504787    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504789    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504792    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504794    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504797    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504801    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504803    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504806    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504808    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504811    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504813    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504816    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504818    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504821    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504824    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504827    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504830    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504833    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504835    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:00.505478 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504837    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504840    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504843    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504845    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504848    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504850    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504853    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504855    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504858    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504860    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504862    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504865    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504867    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504870    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504872    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504874    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504877    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504880    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504882    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504885    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:00.505967 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504888    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504890    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504893    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504895    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504900    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504904    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504907    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504911    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504914    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504917    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504920    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504922    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504925    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504927    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504930    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504932    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504935    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504937    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504939    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504942    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:00.506472 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504944    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:00.506970 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504947    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:00.506970 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504950    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:00.506970 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504952    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:00.506970 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504954    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:00.506970 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504957    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:00.506970 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504960    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:00.506970 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504962    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:00.506970 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504965    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:00.506970 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504967    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:00.506970 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504971    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:00.506970 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504975    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:00.506970 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504977    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:00.506970 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504979    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:00.506970 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.504982    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:00.507604 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507591    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:00.507604 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507604    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507608    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507612    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507614    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507619    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507622    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507625    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507628    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507631    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507634    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507637    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507640    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507642    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507645    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507647    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507650    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507654    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507657    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507661    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:00.507657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507665    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507667    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507670    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507673    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507676    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507679    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507683    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507688    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507691    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507694    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507696    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507699    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507702    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507705    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507707    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507710    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507713    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507715    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507719    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507721    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:00.508151 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507724    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507727    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507731    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507734    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507736    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507739    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507741    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507744    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507746    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507748    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507751    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507753    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507756    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507758    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507762    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507765    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507768    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507771    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507774    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507776    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:00.508682 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507779    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507781    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507784    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507787    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507789    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507792    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507795    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507798    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507800    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507803    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507805    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507808    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507811    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507813    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507815    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507818    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507820    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507822    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507825    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507827    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:00.509212 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507830    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507832    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507835    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507837    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507840    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507842    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507846    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507849    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507851    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507854    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507857    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507880    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507884    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507887    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507890    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507893    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507896    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507899    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.507902    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508318    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:00.509710 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508323    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508326    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508329    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508332    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508335    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508338    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508340    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508343    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508345    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508348    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508351    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508353    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508356    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508358    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508360    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508363    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508366    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508369    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508372    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:00.510214 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508375    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508377    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508380    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508383    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508385    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508388    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508390    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508392    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508395    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508397    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508400    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508402    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508405    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508407    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508410    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508412    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508415    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508417    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508420    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508422    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:00.510684 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508424    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508426    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508429    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508431    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508435    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508439    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508443    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508446    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508448    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508451    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508454    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508457    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508460    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508462    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508466    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508470    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508473    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508476    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508478    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:00.511201 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508481    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508483    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508486    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508488    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508490    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508493    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508495    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508499    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508501    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508504    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508506    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508509    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508512    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508514    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508517    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508519    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508521    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508524    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508526    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508529    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:00.511671 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508532    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508535    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508537    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508540    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508545    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508547    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508550    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508552    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508555    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508557    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508559    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508562    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508565    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508567    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508569    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508572    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508574    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508577    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508580    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.508584    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:00.512176 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508675    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508724    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508731    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508735    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508740    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508744    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508748    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508752    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508756    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508759    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508762    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508766    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508769    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508772    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508775    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508778    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508782    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508785    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508794    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508796    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508799    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508803    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508806    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508814    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:45:00.512668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508817    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508820    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508823    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508826    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508829    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508832    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508834    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508839    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508843    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508846    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508849    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508851    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508854    2561 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508858    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508862    2561 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508866    2561 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508869    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508872    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508875    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508879    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508882    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508884    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508887    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508890    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508894    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:45:00.513282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508897    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508900    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508903    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508906    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508909    2561 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508913    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508916    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508919    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508922    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508925    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508928    2561 flags.go:64] FLAG: --help="false"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508931    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508934    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508937    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508941    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508945    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508948    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508951    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508953    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508956    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508959    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508963    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508966    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508970    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:45:00.513938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508972    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508975    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508978    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508981    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508984    2561 flags.go:64] FLAG: --lock-file=""
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508986    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508989    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508992    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.508998    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509001    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509004    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509007    2561 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509010    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509013    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509016    2561 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509019    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509023    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509026    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509030    2561 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509033    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509036    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509039    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509043    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509047    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509049    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:45:00.514562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509070    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509078    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509081    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509084    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509087    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509090    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509096    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509099    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509102    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509105    2561 flags.go:64] FLAG: --port="10250"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509108    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509111    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a69032fb5a63a016"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509114    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509117    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509119    2561 flags.go:64] FLAG: --register-node="true"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509123    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509127    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509130    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509132    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509135    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509139    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509141    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509145    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509147    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509150    2561 flags.go:64] FLAG: --runonce="false"
Sep 16 18:45:00.515200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509153    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509156    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509158    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509161    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509165    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509168    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509171    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509175    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509177    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509180    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509183    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509186    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509189    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509193    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509196    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509201    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509204    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509207    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509211    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509214    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509217    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509220    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509224    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509227    2561 flags.go:64] FLAG: --v="2"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509231    2561 flags.go:64] FLAG: --version="false"
Sep 16 18:45:00.515846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509235    2561 flags.go:64] FLAG: --vmodule=""
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509239    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.509242    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509362    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509367    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509370    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509373    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509375    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509378    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509381    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509384    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509388    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509391    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509394    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509396    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509399    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509402    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509404    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509407    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:00.516476 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509409    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509412    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509417    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509420    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509424    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509427    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509430    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509433    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509435    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509438    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509442    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509445    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509447    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509450    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509454    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509457    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509460    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509463    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509466    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:00.516939 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509469    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509472    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509475    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509477    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509480    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509484    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509486    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509489    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509492    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509495    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509497    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509500    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509502    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509505    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509507    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509510    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509513    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509517    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509520    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509522    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:00.517422 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509525    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509527    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509530    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509534    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509537    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509539    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509542    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509544    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509547    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509550    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509552    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509555    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509557    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509560    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509563    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509566    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509568    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509572    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509574    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509576    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:00.517926 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509579    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509582    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509585    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509587    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509590    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509593    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509595    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509598    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509600    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509603    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509606    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509608    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509611    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509613    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509616    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509620    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509622    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509625    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509628    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509630    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:00.518437 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509633    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:00.518934 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509635    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:00.518934 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509638    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:00.518934 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.509640    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:00.518934 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.510366    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:00.519306 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.519287    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:45:00.519306 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.519305    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:45:00.520294 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.520285    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:45:00.520331 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.520294    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:45:00.520362 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520355    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:00.520362 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520360    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520364    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520367    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520370    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520373    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520376    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520379    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520381    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520384    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520386    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520389    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520392    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520395    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520398    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520400    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520403    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520407    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520411    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520414    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:00.520414 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520417    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520420    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520423    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520425    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520428    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520431    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520434    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520436    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520439    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520442    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520445    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520447    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520450    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520453    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520455    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520458    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520460    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520462    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520465    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520467    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:00.520867 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520469    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520472    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520474    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520477    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520479    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520482    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520485    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520487    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520490    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520492    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520497    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520500    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520503    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520507    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520509    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520512    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520514    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520517    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520519    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:00.521379 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520522    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520524    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520527    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520530    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520532    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520534    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520537    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520539    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520542    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520545    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520547    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520550    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520552    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520556    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520559    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520562    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520565    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520567    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520570    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520572    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:00.521840 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520575    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520578    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520580    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520583    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520585    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520588    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520591    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520594    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520596    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520598    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520601    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520603    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520605    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520608    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520610    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520613    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520615    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520618    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520621    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:00.522357 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520623    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.520628    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520739    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520743    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520746    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520749    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520752    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520754    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520757    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520760    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520762    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520765    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520767    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520770    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520773    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:00.522848 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520775    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520778    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520780    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520783    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520786    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520789    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520791    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520794    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520796    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520799    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520801    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520804    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520806    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520809    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520811    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520813    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520816    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520818    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520821    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520823    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:00.523231 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520825    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520828    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520830    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520833    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520835    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520837    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520840    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520842    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520845    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520848    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520850    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520853    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520855    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520858    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520861    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520863    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520866    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520869    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520872    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520874    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:00.523759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520878    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520880    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520883    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520887    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520890    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520893    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520896    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520899    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520903    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520906    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520909    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520911    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520914    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520917    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520919    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520922    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520924    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520926    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520929    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:00.524287 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520931    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520934    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520936    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520939    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520941    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520944    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520946    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520950    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520952    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520955    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520957    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520960    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520963    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520965    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520967    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520970    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520972    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520976    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520979    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:00.524745 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520982    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:00.525260 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520984    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:00.525260 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520987    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:00.525260 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520989    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:00.525260 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520992    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:00.525260 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520994    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:00.525260 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520997    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:00.525260 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:00.520999    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:00.525260 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.521004    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:00.525260 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.521140    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:45:00.525260 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.523939    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:45:00.525260 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.524774    2561 server.go:1019] "Starting client certificate rotation"
Sep 16 18:45:00.525260 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.524877    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:00.525260 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.524918    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:00.528229 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.528207    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:00.531268 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.531181    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:00.542446 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.542424    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:45:00.546086 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.546049    2561 log.go:25] "Validated CRI v1 image API"
Sep 16 18:45:00.546938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.546915    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:45:00.550020 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.549994    2561 fs.go:135] Filesystem UUIDs: map[082f3389-aa19-46a9-b2b6-14dcbd87750b:/dev/nvme0n1p3 0dc0e0b8-6ea0-4e6a-8457-f0cffb6ef565:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 16 18:45:00.550111 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.550020    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:45:00.553519 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.553499    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:00.555267 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.555155    2561 manager.go:216] Machine: {Timestamp:2026-09-16 18:45:00.5537792 +0000 UTC m=+0.356497407 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100839 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dd3b951c8393a7fb146b4f79a4159 SystemUUID:ec2dd3b9-51c8-393a-7fb1-46b4f79a4159 BootID:bb25de0a-dd65-4fd8-9254-b58af65bc8e5 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:50:2e:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:50:2e:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:34:a8:1a:b9:f4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:45:00.555267 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.555263    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:45:00.555386 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.555352    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:45:00.558821 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.558794    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:45:00.558977 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.558825    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-125.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:45:00.560451 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.560440    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:45:00.560493 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.560454    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:45:00.560493 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.560463    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:00.561302 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.561291    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:00.562735 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.562724    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:45:00.567396 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.567384    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:00.567523 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.567513    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:45:00.567559 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.567545    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:45:00.567559 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.567556    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:45:00.567619 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.567570    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:45:00.567619 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.567582    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:45:00.567619 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.567592    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:45:00.568039 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.568027    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:00.568098 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.568048    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:00.570489 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.570473    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:45:00.573848 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.573835    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:45:00.573905 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.573867    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:45:00.576969 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.576954    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:45:00.577041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.576974    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:45:00.577041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.576980    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:45:00.577041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.576986    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:45:00.577041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.576994    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:45:00.577041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.576999    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:45:00.577041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.577012    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:45:00.577041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.577020    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:45:00.577041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.577026    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:45:00.577041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.577034    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:45:00.577041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.577044    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:45:00.577322 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.577072    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:45:00.577322 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.577116    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tvz22"
Sep 16 18:45:00.577929 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.577919    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:45:00.577961 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.577930    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:45:00.579039 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.579017    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:45:00.579137 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.579082    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:45:00.581900 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.581886    2561 server.go:1268] "Started kubelet"
Sep 16 18:45:00.582102 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.582001    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:45:00.582207 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.582125    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:45:00.582207 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.582091    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:45:00.582768 ip-10-0-129-125 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:45:00.584376 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.584314    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:45:00.585979 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.585955    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tvz22"
Sep 16 18:45:00.586965 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.586947    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:45:00.588202 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.588140    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:45:00.588681 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.588666    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-125.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:45:00.593639 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.593620    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:00.594435 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.594418    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:45:00.594531 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.594507    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Sep 16 18:45:00.595216 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.595199    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:45:00.595512 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.595494    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:45:00.595593 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.595202    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:45:00.595640 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.595270    2561 factory.go:55] Registering systemd factory
Sep 16 18:45:00.595694 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.595649    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:45:00.595694 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.595688    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:45:00.595771 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.595702    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:45:00.595892 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.595879    2561 factory.go:153] Registering CRI-O factory
Sep 16 18:45:00.595931 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.595898    2561 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:45:00.595979 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.595954    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:45:00.596013 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.595990    2561 factory.go:103] Registering Raw factory
Sep 16 18:45:00.596013 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.596005    2561 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:45:00.596495 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.596481    2561 manager.go:349] Starting recovery of all containers
Sep 16 18:45:00.597857 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.597838    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:00.606914 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.606883    2561 manager.go:354] Recovery completed
Sep 16 18:45:00.607017 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.606989    2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-125.ec2.internal\" not found" node="ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.611042 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.611026    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:00.613099 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.613083    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:00.613181 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.613112    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:00.613181 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.613122    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:00.613688 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.613667    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:45:00.613688 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.613682    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:00.613812 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.613685    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:45:00.613812 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.613718    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:00.614423 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.614410    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:00.614516 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.614440    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:00.614516 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.614456    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:00.615294 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.615282    2561 policy_none.go:49] "None policy: Start"
Sep 16 18:45:00.615338 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.615298    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:45:00.615338 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.615308    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:45:00.616425 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.616415    2561 policy_none.go:47] "Start"
Sep 16 18:45:00.660936 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.660717    2561 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:45:00.675269 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.660980    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:45:00.675269 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.660996    2561 server.go:87] "Starting device plugin registration server"
Sep 16 18:45:00.675269 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.661291    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:45:00.675269 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.661305    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:45:00.675269 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.661480    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:45:00.675269 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.661575    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:45:00.675269 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.661582    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:45:00.675269 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.662157    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:45:00.675269 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.662202    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-125.ec2.internal\" not found"
Sep 16 18:45:00.726198 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.726161    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:45:00.727621 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.727588    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:45:00.727747 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.727658    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:45:00.727747 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.727690    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:45:00.727747 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.727738    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:45:00.729895 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.729873    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:00.762048 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.761977    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:00.762987 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.762968    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:00.763092 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.763001    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:00.763092 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.763012    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:00.763092 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.763035    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.772618 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.772601    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.772685 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.772627    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-125.ec2.internal\": node \"ip-10-0-129-125.ec2.internal\" not found"
Sep 16 18:45:00.786035 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.786010    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Sep 16 18:45:00.828514 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.828474    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"]
Sep 16 18:45:00.828661 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.828578    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:00.829618 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.829602    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:00.829712 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.829634    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:00.829712 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.829649    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:00.831738 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.831723    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:00.831865 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.831847    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.831926 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.831889    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:00.832480 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.832465    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:00.832480 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.832472    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:00.832585 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.832491    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:00.832585 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.832497    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:00.832585 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.832503    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:00.832585 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.832512    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:00.834535 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.834521    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.834610 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.834551    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:00.835315 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.835300    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:00.835397 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.835327    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:00.835397 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.835336    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:00.861831 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.861809    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-125.ec2.internal\" not found" node="ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.866437 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.866416    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-125.ec2.internal\" not found" node="ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.887112 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.887087    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Sep 16 18:45:00.896840 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.896816    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d69cc83b62a0c0df434d27c89a80e90-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"3d69cc83b62a0c0df434d27c89a80e90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.896957 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.896850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d69cc83b62a0c0df434d27c89a80e90-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"3d69cc83b62a0c0df434d27c89a80e90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.896957 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.896868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7de017d6d245b795fdd5ece3d22237c-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.ec2.internal\" (UID: \"c7de017d6d245b795fdd5ece3d22237c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.987708 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:00.987677    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Sep 16 18:45:00.997090 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.997033    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d69cc83b62a0c0df434d27c89a80e90-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"3d69cc83b62a0c0df434d27c89a80e90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.997240 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.997121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7de017d6d245b795fdd5ece3d22237c-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.ec2.internal\" (UID: \"c7de017d6d245b795fdd5ece3d22237c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.997240 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.997132    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d69cc83b62a0c0df434d27c89a80e90-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"3d69cc83b62a0c0df434d27c89a80e90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.997240 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.997147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d69cc83b62a0c0df434d27c89a80e90-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"3d69cc83b62a0c0df434d27c89a80e90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.997240 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.997193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d69cc83b62a0c0df434d27c89a80e90-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"3d69cc83b62a0c0df434d27c89a80e90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:00.997240 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:00.997198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7de017d6d245b795fdd5ece3d22237c-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.ec2.internal\" (UID: \"c7de017d6d245b795fdd5ece3d22237c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:01.088515 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.088416    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Sep 16 18:45:01.165942 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.165898    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:01.171443 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.171300    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:01.188655 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.188632    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Sep 16 18:45:01.289251 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.289214    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Sep 16 18:45:01.389842 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.389765    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Sep 16 18:45:01.467367 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.467332    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:01.495342 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.495316    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:01.505720 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.505696    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:01.507587 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.507570    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Sep 16 18:45:01.519124 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.519102    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:01.525258 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.525240    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:45:01.525367 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.525350    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:45:01.525411 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.525380    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:45:01.525450 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.525400    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:45:01.568594 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.568562    2561 apiserver.go:52] "Watching apiserver"
Sep 16 18:45:01.586523 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.586491    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:45:01.586901 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.586876    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-hm84r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal","openshift-multus/multus-4vw5f","openshift-multus/multus-additional-cni-plugins-ck2ml","openshift-network-diagnostics/network-check-target-vmv47","openshift-network-operator/iptables-alerter-q79dm","kube-system/konnectivity-agent-lvn8l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw","openshift-cluster-node-tuning-operator/tuned-lrj6j","openshift-image-registry/node-ca-ldqrg","openshift-multus/network-metrics-daemon-65n74","openshift-ovn-kubernetes/ovnkube-node-chgz7","kube-system/global-pull-secret-syncer-l6stq","kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"]
Sep 16 18:45:01.589193 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.589171    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lvn8l"
Sep 16 18:45:01.591136 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.591121    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:01.591199 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.591185    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:01.592330 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.592312    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7gjdq\""
Sep 16 18:45:01.592436 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.592312    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:45:01.592817 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.592420    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:40:01 +0000 UTC" deadline="2028-02-19 00:42:56.981228011 +0000 UTC"
Sep 16 18:45:01.592817 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.592446    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12485h57m55.388784699s"
Sep 16 18:45:01.592817 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.592611    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:45:01.593922 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.593408    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.593922 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.593809    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:01.596537 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.596511    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:45:01.596631 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.596546    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.597340 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.597324    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-chxht\""
Sep 16 18:45:01.597811 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.597795    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:45:01.597861 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.597843    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:45:01.597906 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.597853    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:45:01.598719 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.598701    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:01.598806 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.598753    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:01.599639 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.599613    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-system-cni-dir\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.599762 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.599653    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3087807d-e287-4a6d-b99c-17aa3ab3e6a5-agent-certs\") pod \"konnectivity-agent-lvn8l\" (UID: \"3087807d-e287-4a6d-b99c-17aa3ab3e6a5\") " pod="kube-system/konnectivity-agent-lvn8l"
Sep 16 18:45:01.599762 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.599681    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3087807d-e287-4a6d-b99c-17aa3ab3e6a5-konnectivity-ca\") pod \"konnectivity-agent-lvn8l\" (UID: \"3087807d-e287-4a6d-b99c-17aa3ab3e6a5\") " pod="kube-system/konnectivity-agent-lvn8l"
Sep 16 18:45:01.599762 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.599748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-dbus\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:01.599871 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.599773    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1599ebeb-4855-44c5-848f-b381ca2fb434-cni-binary-copy\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.599871 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.599787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-var-lib-cni-bin\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.599871 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.599830    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-run-multus-certs\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.599871 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.599846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-etc-kubernetes\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.599871 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.599862    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-run-netns\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.600033 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.599907    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-cnibin\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.600033 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.599932    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-cnibin\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.600033 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.599949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.600033 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.599975    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-var-lib-cni-multus\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.600033 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.599991    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-var-lib-kubelet\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.600033 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600004    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-system-cni-dir\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.600033 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-hostroot\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.600276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600041    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.600276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-multus-socket-dir-parent\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.600276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-run-k8s-cni-cncf-io\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.600276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600141    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-multus-conf-dir\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.600276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600162    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px97v\" (UniqueName: \"kubernetes.io/projected/1599ebeb-4855-44c5-848f-b381ca2fb434-kube-api-access-px97v\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.600276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.600276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600206    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-cni-binary-copy\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.600276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600230    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-kubelet-config\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:01.600276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600246    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:01.600276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600262    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-multus-cni-dir\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.600276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600276    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1599ebeb-4855-44c5-848f-b381ca2fb434-multus-daemon-config\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.600731 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxnlb\" (UniqueName: \"kubernetes.io/projected/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-kube-api-access-cxnlb\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.600731 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600557    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gxfwj\""
Sep 16 18:45:01.600731 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600557    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:45:01.600731 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.600609    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:45:01.601100 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.601080    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q79dm"
Sep 16 18:45:01.603444 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.603427    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.604376 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.604357    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:01.604575 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.604549    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:45:01.604790 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.604774    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:01.605044 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.605027    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fgdr2\""
Sep 16 18:45:01.605948 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.605932    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hm84r"
Sep 16 18:45:01.607341 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.607324    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:01.607598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.607584    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8k2wh\""
Sep 16 18:45:01.607662 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.607584    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:01.608493 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.608476    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.610990 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.610971    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ldqrg"
Sep 16 18:45:01.613454 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.613438    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:01.613536 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.613522    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:01.614365 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.614349    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:45:01.614771 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.614758    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:45:01.614978 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.614965    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cndmf\""
Sep 16 18:45:01.615092 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.615054    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:45:01.615662 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.615643    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.615844 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.615823    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xj8nj\""
Sep 16 18:45:01.616150 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.616135    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g9rmr\""
Sep 16 18:45:01.616331 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.616311    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:45:01.616403 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.616381    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:45:01.616456 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.616412    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:45:01.617048 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.617033    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:45:01.617093 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.617082    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:45:01.618745 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.618727    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:45:01.618841 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.618826    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:45:01.618888 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.618858    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:45:01.618969 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.618954    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:45:01.619346 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.619331    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:45:01.619737 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.619717    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t8xlg\""
Sep 16 18:45:01.620889 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.620871    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:01.620982 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.620888    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:45:01.638930 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.638904    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hnvfl"
Sep 16 18:45:01.649689 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.649667    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hnvfl"
Sep 16 18:45:01.697368 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.697343    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:45:01.700837 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.700817    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-var-lib-cni-multus\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.700961 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.700856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bmp9\" (UniqueName: \"kubernetes.io/projected/5e84101c-123e-475f-ad77-fbe61af69584-kube-api-access-8bmp9\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.700961 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.700888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-var-lib-kubelet\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.700961 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.700913    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-tuned\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.700961 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.700923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-var-lib-cni-multus\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.700961 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.700938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5223494-bd2b-4911-ae88-8bb10b49b6f5-host\") pod \"node-ca-ldqrg\" (UID: \"a5223494-bd2b-4911-ae88-8bb10b49b6f5\") " pod="openshift-image-registry/node-ca-ldqrg"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.700976    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcwpn\" (UniqueName: \"kubernetes.io/projected/a5223494-bd2b-4911-ae88-8bb10b49b6f5-kube-api-access-mcwpn\") pod \"node-ca-ldqrg\" (UID: \"a5223494-bd2b-4911-ae88-8bb10b49b6f5\") " pod="openshift-image-registry/node-ca-ldqrg"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-system-cni-dir\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701023    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5e84101c-123e-475f-ad77-fbe61af69584-ovnkube-script-lib\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701048    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv9w5\" (UniqueName: \"kubernetes.io/projected/7318780a-dbbe-4905-9e89-ce48d61c0d30-kube-api-access-fv9w5\") pod \"iptables-alerter-q79dm\" (UID: \"7318780a-dbbe-4905-9e89-ce48d61c0d30\") " pod="openshift-network-operator/iptables-alerter-q79dm"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zmw\" (UniqueName: \"kubernetes.io/projected/7d21675b-89e7-4f54-8340-6228f09668e9-kube-api-access-r5zmw\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-kubelet\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701108    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-system-cni-dir\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-systemd-units\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-run-k8s-cni-cncf-io\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701234    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px97v\" (UniqueName: \"kubernetes.io/projected/1599ebeb-4855-44c5-848f-b381ca2fb434-kube-api-access-px97v\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-sysctl-conf\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701292    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-slash\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701293    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-run-k8s-cni-cncf-io\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701317    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-cni-bin\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.702325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701356    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5e84101c-123e-475f-ad77-fbe61af69584-ovnkube-config\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e6040a3-2386-4096-90e7-fb3bec2c2f60-tmp\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-multus-cni-dir\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1599ebeb-4855-44c5-848f-b381ca2fb434-multus-daemon-config\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxnlb\" (UniqueName: \"kubernetes.io/projected/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-kube-api-access-cxnlb\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.701598    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-run-systemd\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-multus-cni-dir\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7318780a-dbbe-4905-9e89-ce48d61c0d30-host-slash\") pod \"iptables-alerter-q79dm\" (UID: \"7318780a-dbbe-4905-9e89-ce48d61c0d30\") " pod="openshift-network-operator/iptables-alerter-q79dm"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701758    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slf4b\" (UniqueName: \"kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b\") pod \"network-check-target-vmv47\" (UID: \"f247b77e-176f-4657-9447-ef0b89f708b3\") " pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.701795    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret podName:0c7a47e0-a2e8-4c1d-815b-18c9cecea99d nodeName:}" failed. No retries permitted until 2026-09-16 18:45:02.201754217 +0000 UTC m=+2.004472431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret") pod "global-pull-secret-syncer-l6stq" (UID: "0c7a47e0-a2e8-4c1d-815b-18c9cecea99d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701837    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-run\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3087807d-e287-4a6d-b99c-17aa3ab3e6a5-konnectivity-ca\") pod \"konnectivity-agent-lvn8l\" (UID: \"3087807d-e287-4a6d-b99c-17aa3ab3e6a5\") " pod="kube-system/konnectivity-agent-lvn8l"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-dbus\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701916    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1599ebeb-4855-44c5-848f-b381ca2fb434-cni-binary-copy\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.702971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701940    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-etc-kubernetes\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-log-socket\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-dbus\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-systemd\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.701991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-etc-kubernetes\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702013    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-lib-modules\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702059    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-run-netns\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1599ebeb-4855-44c5-848f-b381ca2fb434-multus-daemon-config\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7318780a-dbbe-4905-9e89-ce48d61c0d30-iptables-alerter-script\") pod \"iptables-alerter-q79dm\" (UID: \"7318780a-dbbe-4905-9e89-ce48d61c0d30\") " pod="openshift-network-operator/iptables-alerter-q79dm"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-cnibin\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702170    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-cnibin\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702195    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r82j\" (UniqueName: \"kubernetes.io/projected/6b8c5f93-a582-446f-873c-5b3344f1c2c5-kube-api-access-5r82j\") pod \"network-metrics-daemon-65n74\" (UID: \"6b8c5f93-a582-446f-873c-5b3344f1c2c5\") " pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-var-lib-openvswitch\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-cnibin\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5e84101c-123e-475f-ad77-fbe61af69584-env-overrides\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702272    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-cnibin\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.703598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702285    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-registration-dir\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702331    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-var-lib-kubelet\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-node-log\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-cni-netd\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702374    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-var-lib-kubelet\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702393    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-sysconfig\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702428    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-sys\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702433    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3087807d-e287-4a6d-b99c-17aa3ab3e6a5-konnectivity-ca\") pod \"konnectivity-agent-lvn8l\" (UID: \"3087807d-e287-4a6d-b99c-17aa3ab3e6a5\") " pod="kube-system/konnectivity-agent-lvn8l"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702454    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-socket-dir\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-hostroot\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1599ebeb-4855-44c5-848f-b381ca2fb434-cni-binary-copy\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702521    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e84101c-123e-475f-ad77-fbe61af69584-ovn-node-metrics-cert\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702546    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a5223494-bd2b-4911-ae88-8bb10b49b6f5-serviceca\") pod \"node-ca-ldqrg\" (UID: \"a5223494-bd2b-4911-ae88-8bb10b49b6f5\") " pod="openshift-image-registry/node-ca-ldqrg"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702572    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-run-openvswitch\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702524    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-hostroot\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-multus-socket-dir-parent\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.704225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-multus-conf-dir\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs\") pod \"network-metrics-daemon-65n74\" (UID: \"6b8c5f93-a582-446f-873c-5b3344f1c2c5\") " pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702728    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-multus-conf-dir\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702718    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-multus-socket-dir-parent\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702754    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-host\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702754    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702813    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-sys-fs\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-sysctl-d\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ca39171-efab-4995-a0fb-2261a074d3c9-hosts-file\") pod \"node-resolver-hm84r\" (UID: \"0ca39171-efab-4995-a0fb-2261a074d3c9\") " pod="openshift-dns/node-resolver-hm84r"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702878    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-cni-binary-copy\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-kubelet-config\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-etc-openvswitch\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.702970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-run-ovn-kubernetes\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-kubelet-config\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703078    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-system-cni-dir\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.704780 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703115    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3087807d-e287-4a6d-b99c-17aa3ab3e6a5-agent-certs\") pod \"konnectivity-agent-lvn8l\" (UID: \"3087807d-e287-4a6d-b99c-17aa3ab3e6a5\") " pod="kube-system/konnectivity-agent-lvn8l"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-var-lib-cni-bin\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703170    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-run-multus-certs\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctwcj\" (UniqueName: \"kubernetes.io/projected/1e6040a3-2386-4096-90e7-fb3bec2c2f60-kube-api-access-ctwcj\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703226    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v95t\" (UniqueName: \"kubernetes.io/projected/0ca39171-efab-4995-a0fb-2261a074d3c9-kube-api-access-9v95t\") pod \"node-resolver-hm84r\" (UID: \"0ca39171-efab-4995-a0fb-2261a074d3c9\") " pod="openshift-dns/node-resolver-hm84r"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703243    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-run-netns\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703247    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-cni-binary-copy\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-run-ovn\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703292    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-modprobe-d\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703322    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-kubernetes\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703344    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-device-dir\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703398    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca39171-efab-4995-a0fb-2261a074d3c9-tmp-dir\") pod \"node-resolver-hm84r\" (UID: \"0ca39171-efab-4995-a0fb-2261a074d3c9\") " pod="openshift-dns/node-resolver-hm84r"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703447    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-system-cni-dir\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703470    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-run-netns\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703526    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-run-multus-certs\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703565    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1599ebeb-4855-44c5-848f-b381ca2fb434-host-var-lib-cni-bin\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.705333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703828    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:45:01.705871 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.703937    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.707782 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.707763    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3087807d-e287-4a6d-b99c-17aa3ab3e6a5-agent-certs\") pod \"konnectivity-agent-lvn8l\" (UID: \"3087807d-e287-4a6d-b99c-17aa3ab3e6a5\") " pod="kube-system/konnectivity-agent-lvn8l"
Sep 16 18:45:01.713565 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.713542    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px97v\" (UniqueName: \"kubernetes.io/projected/1599ebeb-4855-44c5-848f-b381ca2fb434-kube-api-access-px97v\") pod \"multus-4vw5f\" (UID: \"1599ebeb-4855-44c5-848f-b381ca2fb434\") " pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.713665 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.713605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxnlb\" (UniqueName: \"kubernetes.io/projected/7ca4a1da-ce89-4def-9f1f-01f4ea36b08d-kube-api-access-cxnlb\") pod \"multus-additional-cni-plugins-ck2ml\" (UID: \"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d\") " pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804002    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-systemd-units\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-sysctl-conf\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804118    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-slash\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804159    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-cni-bin\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804239    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5e84101c-123e-475f-ad77-fbe61af69584-ovnkube-config\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-slash\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804364    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e6040a3-2386-4096-90e7-fb3bec2c2f60-tmp\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804419    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-run-systemd\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7318780a-dbbe-4905-9e89-ce48d61c0d30-host-slash\") pod \"iptables-alerter-q79dm\" (UID: \"7318780a-dbbe-4905-9e89-ce48d61c0d30\") " pod="openshift-network-operator/iptables-alerter-q79dm"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804493    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slf4b\" (UniqueName: \"kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b\") pod \"network-check-target-vmv47\" (UID: \"f247b77e-176f-4657-9447-ef0b89f708b3\") " pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-run\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-log-socket\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-systemd\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-lib-modules\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804650    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-run-netns\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.806266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7318780a-dbbe-4905-9e89-ce48d61c0d30-iptables-alerter-script\") pod \"iptables-alerter-q79dm\" (UID: \"7318780a-dbbe-4905-9e89-ce48d61c0d30\") " pod="openshift-network-operator/iptables-alerter-q79dm"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r82j\" (UniqueName: \"kubernetes.io/projected/6b8c5f93-a582-446f-873c-5b3344f1c2c5-kube-api-access-5r82j\") pod \"network-metrics-daemon-65n74\" (UID: \"6b8c5f93-a582-446f-873c-5b3344f1c2c5\") " pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804747    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-var-lib-openvswitch\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804779    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5e84101c-123e-475f-ad77-fbe61af69584-env-overrides\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804817    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-registration-dir\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-node-log\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-cni-netd\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804903    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5e84101c-123e-475f-ad77-fbe61af69584-ovnkube-config\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-sysconfig\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804963    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-sys\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804984    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-sysconfig\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.804995    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-socket-dir\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805033    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e84101c-123e-475f-ad77-fbe61af69584-ovn-node-metrics-cert\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-systemd-units\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a5223494-bd2b-4911-ae88-8bb10b49b6f5-serviceca\") pod \"node-ca-ldqrg\" (UID: \"a5223494-bd2b-4911-ae88-8bb10b49b6f5\") " pod="openshift-image-registry/node-ca-ldqrg"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805099    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-run-openvswitch\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs\") pod \"network-metrics-daemon-65n74\" (UID: \"6b8c5f93-a582-446f-873c-5b3344f1c2c5\") " pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:01.807347 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805193    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-host\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805227    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-sysctl-conf\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-sys-fs\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805291    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-sysctl-d\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ca39171-efab-4995-a0fb-2261a074d3c9-hosts-file\") pod \"node-resolver-hm84r\" (UID: \"0ca39171-efab-4995-a0fb-2261a074d3c9\") " pod="openshift-dns/node-resolver-hm84r"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-etc-openvswitch\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805427    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-run-ovn-kubernetes\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctwcj\" (UniqueName: \"kubernetes.io/projected/1e6040a3-2386-4096-90e7-fb3bec2c2f60-kube-api-access-ctwcj\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v95t\" (UniqueName: \"kubernetes.io/projected/0ca39171-efab-4995-a0fb-2261a074d3c9-kube-api-access-9v95t\") pod \"node-resolver-hm84r\" (UID: \"0ca39171-efab-4995-a0fb-2261a074d3c9\") " pod="openshift-dns/node-resolver-hm84r"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-run-ovn\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-modprobe-d\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-kubernetes\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-device-dir\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca39171-efab-4995-a0fb-2261a074d3c9-tmp-dir\") pod \"node-resolver-hm84r\" (UID: \"0ca39171-efab-4995-a0fb-2261a074d3c9\") " pod="openshift-dns/node-resolver-hm84r"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805694    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bmp9\" (UniqueName: \"kubernetes.io/projected/5e84101c-123e-475f-ad77-fbe61af69584-kube-api-access-8bmp9\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.808220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805726    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-var-lib-kubelet\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-tuned\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805790    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5223494-bd2b-4911-ae88-8bb10b49b6f5-host\") pod \"node-ca-ldqrg\" (UID: \"a5223494-bd2b-4911-ae88-8bb10b49b6f5\") " pod="openshift-image-registry/node-ca-ldqrg"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805826    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcwpn\" (UniqueName: \"kubernetes.io/projected/a5223494-bd2b-4911-ae88-8bb10b49b6f5-kube-api-access-mcwpn\") pod \"node-ca-ldqrg\" (UID: \"a5223494-bd2b-4911-ae88-8bb10b49b6f5\") " pod="openshift-image-registry/node-ca-ldqrg"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5e84101c-123e-475f-ad77-fbe61af69584-ovnkube-script-lib\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv9w5\" (UniqueName: \"kubernetes.io/projected/7318780a-dbbe-4905-9e89-ce48d61c0d30-kube-api-access-fv9w5\") pod \"iptables-alerter-q79dm\" (UID: \"7318780a-dbbe-4905-9e89-ce48d61c0d30\") " pod="openshift-network-operator/iptables-alerter-q79dm"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805928    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zmw\" (UniqueName: \"kubernetes.io/projected/7d21675b-89e7-4f54-8340-6228f09668e9-kube-api-access-r5zmw\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.805961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-kubelet\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806074    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-kubelet\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-cni-bin\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806205    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806344    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-cni-netd\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806361    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-var-lib-openvswitch\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806592    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-kubernetes\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806594    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-device-dir\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806621    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-run-ovn\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-run-netns\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.808816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-systemd\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806710    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-run-openvswitch\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.806756    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806761    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5223494-bd2b-4911-ae88-8bb10b49b6f5-host\") pod \"node-ca-ldqrg\" (UID: \"a5223494-bd2b-4911-ae88-8bb10b49b6f5\") " pod="openshift-image-registry/node-ca-ldqrg"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-log-socket\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5e84101c-123e-475f-ad77-fbe61af69584-env-overrides\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.806824    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs podName:6b8c5f93-a582-446f-873c-5b3344f1c2c5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:02.306803515 +0000 UTC m=+2.109521731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs") pod "network-metrics-daemon-65n74" (UID: "6b8c5f93-a582-446f-873c-5b3344f1c2c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806859    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-host-run-ovn-kubernetes\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806917    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-lib-modules\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-sys\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806916    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-sysctl-d\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca39171-efab-4995-a0fb-2261a074d3c9-tmp-dir\") pod \"node-resolver-hm84r\" (UID: \"0ca39171-efab-4995-a0fb-2261a074d3c9\") " pod="openshift-dns/node-resolver-hm84r"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.806999    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-etc-openvswitch\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.807024    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ca39171-efab-4995-a0fb-2261a074d3c9-hosts-file\") pod \"node-resolver-hm84r\" (UID: \"0ca39171-efab-4995-a0fb-2261a074d3c9\") " pod="openshift-dns/node-resolver-hm84r"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.807057    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-registration-dir\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.807172    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.807181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-modprobe-d\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.807248    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-var-lib-kubelet\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.809469 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.807330    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-node-log\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.810187 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.807513    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-run\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.810187 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.807576    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7318780a-dbbe-4905-9e89-ce48d61c0d30-host-slash\") pod \"iptables-alerter-q79dm\" (UID: \"7318780a-dbbe-4905-9e89-ce48d61c0d30\") " pod="openshift-network-operator/iptables-alerter-q79dm"
Sep 16 18:45:01.810187 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.807643    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5e84101c-123e-475f-ad77-fbe61af69584-run-systemd\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.810187 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.807027    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.810187 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.807723    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-sys-fs\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.810187 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.807740    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7318780a-dbbe-4905-9e89-ce48d61c0d30-iptables-alerter-script\") pod \"iptables-alerter-q79dm\" (UID: \"7318780a-dbbe-4905-9e89-ce48d61c0d30\") " pod="openshift-network-operator/iptables-alerter-q79dm"
Sep 16 18:45:01.810187 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.807981    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7d21675b-89e7-4f54-8340-6228f09668e9-socket-dir\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.810187 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.807250    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e6040a3-2386-4096-90e7-fb3bec2c2f60-host\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.810187 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.808740    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e6040a3-2386-4096-90e7-fb3bec2c2f60-tmp\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.810187 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.808822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a5223494-bd2b-4911-ae88-8bb10b49b6f5-serviceca\") pod \"node-ca-ldqrg\" (UID: \"a5223494-bd2b-4911-ae88-8bb10b49b6f5\") " pod="openshift-image-registry/node-ca-ldqrg"
Sep 16 18:45:01.810187 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.809308    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5e84101c-123e-475f-ad77-fbe61af69584-ovnkube-script-lib\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.810187 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.809813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e84101c-123e-475f-ad77-fbe61af69584-ovn-node-metrics-cert\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.810622 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.810530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e6040a3-2386-4096-90e7-fb3bec2c2f60-etc-tuned\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.816768 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.816747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zmw\" (UniqueName: \"kubernetes.io/projected/7d21675b-89e7-4f54-8340-6228f09668e9-kube-api-access-r5zmw\") pod \"aws-ebs-csi-driver-node-gkhpw\" (UID: \"7d21675b-89e7-4f54-8340-6228f09668e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:01.818827 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.818533    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:01.818827 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.818564    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:01.818827 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.818579    2561 projected.go:196] Error preparing data for projected volume kube-api-access-slf4b for pod openshift-network-diagnostics/network-check-target-vmv47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:01.818827 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:01.818659    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b podName:f247b77e-176f-4657-9447-ef0b89f708b3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:02.318639108 +0000 UTC m=+2.121357319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-slf4b" (UniqueName: "kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b") pod "network-check-target-vmv47" (UID: "f247b77e-176f-4657-9447-ef0b89f708b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:01.820805 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.820775    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv9w5\" (UniqueName: \"kubernetes.io/projected/7318780a-dbbe-4905-9e89-ce48d61c0d30-kube-api-access-fv9w5\") pod \"iptables-alerter-q79dm\" (UID: \"7318780a-dbbe-4905-9e89-ce48d61c0d30\") " pod="openshift-network-operator/iptables-alerter-q79dm"
Sep 16 18:45:01.821288 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.821258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v95t\" (UniqueName: \"kubernetes.io/projected/0ca39171-efab-4995-a0fb-2261a074d3c9-kube-api-access-9v95t\") pod \"node-resolver-hm84r\" (UID: \"0ca39171-efab-4995-a0fb-2261a074d3c9\") " pod="openshift-dns/node-resolver-hm84r"
Sep 16 18:45:01.821673 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.821651    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctwcj\" (UniqueName: \"kubernetes.io/projected/1e6040a3-2386-4096-90e7-fb3bec2c2f60-kube-api-access-ctwcj\") pod \"tuned-lrj6j\" (UID: \"1e6040a3-2386-4096-90e7-fb3bec2c2f60\") " pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.821745 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.821670    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bmp9\" (UniqueName: \"kubernetes.io/projected/5e84101c-123e-475f-ad77-fbe61af69584-kube-api-access-8bmp9\") pod \"ovnkube-node-chgz7\" (UID: \"5e84101c-123e-475f-ad77-fbe61af69584\") " pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:01.821745 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.821674    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r82j\" (UniqueName: \"kubernetes.io/projected/6b8c5f93-a582-446f-873c-5b3344f1c2c5-kube-api-access-5r82j\") pod \"network-metrics-daemon-65n74\" (UID: \"6b8c5f93-a582-446f-873c-5b3344f1c2c5\") " pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:01.821931 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.821915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcwpn\" (UniqueName: \"kubernetes.io/projected/a5223494-bd2b-4911-ae88-8bb10b49b6f5-kube-api-access-mcwpn\") pod \"node-ca-ldqrg\" (UID: \"a5223494-bd2b-4911-ae88-8bb10b49b6f5\") " pod="openshift-image-registry/node-ca-ldqrg"
Sep 16 18:45:01.912180 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.912047    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lvn8l"
Sep 16 18:45:01.927045 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.927011    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4vw5f"
Sep 16 18:45:01.944286 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.944241    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ck2ml"
Sep 16 18:45:01.949295 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.949274    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q79dm"
Sep 16 18:45:01.954247 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.954227    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lrj6j"
Sep 16 18:45:01.982377 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:01.982353    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hm84r"
Sep 16 18:45:02.004273 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.004246    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw"
Sep 16 18:45:02.027323 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.027298    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:02.029766 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.029739    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ldqrg"
Sep 16 18:45:02.034932 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.034896    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:02.186934 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:02.186897    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d69cc83b62a0c0df434d27c89a80e90.slice/crio-5d45b3471ee6f4720abaa9e34c0b5c7385ee179463addc99313a4a38b56d9a99 WatchSource:0}: Error finding container 5d45b3471ee6f4720abaa9e34c0b5c7385ee179463addc99313a4a38b56d9a99: Status 404 returned error can't find the container with id 5d45b3471ee6f4720abaa9e34c0b5c7385ee179463addc99313a4a38b56d9a99
Sep 16 18:45:02.189707 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:02.189657    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7de017d6d245b795fdd5ece3d22237c.slice/crio-3fe3f95e24d30c18dba393be3243274bf188d4b90266410263006baaa8499df7 WatchSource:0}: Error finding container 3fe3f95e24d30c18dba393be3243274bf188d4b90266410263006baaa8499df7: Status 404 returned error can't find the container with id 3fe3f95e24d30c18dba393be3243274bf188d4b90266410263006baaa8499df7
Sep 16 18:45:02.195165 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.195142    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:45:02.209101 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.209058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:02.209239 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:02.209220    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:02.209291 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:02.209280    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret podName:0c7a47e0-a2e8-4c1d-815b-18c9cecea99d nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.209264479 +0000 UTC m=+3.011982675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret") pod "global-pull-secret-syncer-l6stq" (UID: "0c7a47e0-a2e8-4c1d-815b-18c9cecea99d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:02.302395 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:02.302367    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1599ebeb_4855_44c5_848f_b381ca2fb434.slice/crio-caaab697507d9cc5c7e5a0375c488b7229a2e900a5cd813f3eb729389ccdf648 WatchSource:0}: Error finding container caaab697507d9cc5c7e5a0375c488b7229a2e900a5cd813f3eb729389ccdf648: Status 404 returned error can't find the container with id caaab697507d9cc5c7e5a0375c488b7229a2e900a5cd813f3eb729389ccdf648
Sep 16 18:45:02.309774 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.309753    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs\") pod \"network-metrics-daemon-65n74\" (UID: \"6b8c5f93-a582-446f-873c-5b3344f1c2c5\") " pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:02.309909 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:02.309894    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:02.309976 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:02.309964    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs podName:6b8c5f93-a582-446f-873c-5b3344f1c2c5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.309944272 +0000 UTC m=+3.112662670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs") pod "network-metrics-daemon-65n74" (UID: "6b8c5f93-a582-446f-873c-5b3344f1c2c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:02.410726 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.410690    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slf4b\" (UniqueName: \"kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b\") pod \"network-check-target-vmv47\" (UID: \"f247b77e-176f-4657-9447-ef0b89f708b3\") " pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:02.410897 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:02.410834    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:02.410897 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:02.410851    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:02.410897 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:02.410861    2561 projected.go:196] Error preparing data for projected volume kube-api-access-slf4b for pod openshift-network-diagnostics/network-check-target-vmv47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:02.411007 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:02.410910    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b podName:f247b77e-176f-4657-9447-ef0b89f708b3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.410896051 +0000 UTC m=+3.213614245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-slf4b" (UniqueName: "kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b") pod "network-check-target-vmv47" (UID: "f247b77e-176f-4657-9447-ef0b89f708b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:02.454822 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:02.454793    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d21675b_89e7_4f54_8340_6228f09668e9.slice/crio-59e68184d8d0455844c401fc6601740db42e327716f78034348cc899321f9b46 WatchSource:0}: Error finding container 59e68184d8d0455844c401fc6601740db42e327716f78034348cc899321f9b46: Status 404 returned error can't find the container with id 59e68184d8d0455844c401fc6601740db42e327716f78034348cc899321f9b46
Sep 16 18:45:02.491946 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:02.491897    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6040a3_2386_4096_90e7_fb3bec2c2f60.slice/crio-58cdcb4f694688b62d18d277388e0dd4afb2650be74bd176bde029b36fb4ca26 WatchSource:0}: Error finding container 58cdcb4f694688b62d18d277388e0dd4afb2650be74bd176bde029b36fb4ca26: Status 404 returned error can't find the container with id 58cdcb4f694688b62d18d277388e0dd4afb2650be74bd176bde029b36fb4ca26
Sep 16 18:45:02.494557 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:02.494111    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca4a1da_ce89_4def_9f1f_01f4ea36b08d.slice/crio-df6ddb608719c61e8b844e83c082d8a20c2b3006b8bbb1ba070a4ad00d727566 WatchSource:0}: Error finding container df6ddb608719c61e8b844e83c082d8a20c2b3006b8bbb1ba070a4ad00d727566: Status 404 returned error can't find the container with id df6ddb608719c61e8b844e83c082d8a20c2b3006b8bbb1ba070a4ad00d727566
Sep 16 18:45:02.528244 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:02.528217    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5223494_bd2b_4911_ae88_8bb10b49b6f5.slice/crio-3f857a92decce35c762072463303b72a975ae47dccf57f0364ccb565a6b151f3 WatchSource:0}: Error finding container 3f857a92decce35c762072463303b72a975ae47dccf57f0364ccb565a6b151f3: Status 404 returned error can't find the container with id 3f857a92decce35c762072463303b72a975ae47dccf57f0364ccb565a6b151f3
Sep 16 18:45:02.588519 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:02.588489    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e84101c_123e_475f_ad77_fbe61af69584.slice/crio-6501049d95280b4ac3492118d2a5140a5a51e6b03a0c4b0364a815f47dcd3817 WatchSource:0}: Error finding container 6501049d95280b4ac3492118d2a5140a5a51e6b03a0c4b0364a815f47dcd3817: Status 404 returned error can't find the container with id 6501049d95280b4ac3492118d2a5140a5a51e6b03a0c4b0364a815f47dcd3817
Sep 16 18:45:02.596235 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:02.596199    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7318780a_dbbe_4905_9e89_ce48d61c0d30.slice/crio-6907cadbd5f77cd4f99d059895eaa31b04e5f362da905cd24e4156cbaf8a4620 WatchSource:0}: Error finding container 6907cadbd5f77cd4f99d059895eaa31b04e5f362da905cd24e4156cbaf8a4620: Status 404 returned error can't find the container with id 6907cadbd5f77cd4f99d059895eaa31b04e5f362da905cd24e4156cbaf8a4620
Sep 16 18:45:02.623783 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.623759    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:02.635336 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.635310    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:02.652967 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.652912    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:02 +0000 UTC" deadline="2028-02-14 03:32:51.894884058 +0000 UTC"
Sep 16 18:45:02.652967 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.652960    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12368h47m49.241927998s"
Sep 16 18:45:02.735979 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.735912    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q79dm" event={"ID":"7318780a-dbbe-4905-9e89-ce48d61c0d30","Type":"ContainerStarted","Data":"6907cadbd5f77cd4f99d059895eaa31b04e5f362da905cd24e4156cbaf8a4620"}
Sep 16 18:45:02.737443 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.737397    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7" event={"ID":"5e84101c-123e-475f-ad77-fbe61af69584","Type":"ContainerStarted","Data":"6501049d95280b4ac3492118d2a5140a5a51e6b03a0c4b0364a815f47dcd3817"}
Sep 16 18:45:02.739004 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.738975    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ck2ml" event={"ID":"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d","Type":"ContainerStarted","Data":"df6ddb608719c61e8b844e83c082d8a20c2b3006b8bbb1ba070a4ad00d727566"}
Sep 16 18:45:02.740418 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.740387    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw" event={"ID":"7d21675b-89e7-4f54-8340-6228f09668e9","Type":"ContainerStarted","Data":"59e68184d8d0455844c401fc6601740db42e327716f78034348cc899321f9b46"}
Sep 16 18:45:02.741749 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.741714    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" event={"ID":"3d69cc83b62a0c0df434d27c89a80e90","Type":"ContainerStarted","Data":"5d45b3471ee6f4720abaa9e34c0b5c7385ee179463addc99313a4a38b56d9a99"}
Sep 16 18:45:02.743269 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.743235    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ldqrg" event={"ID":"a5223494-bd2b-4911-ae88-8bb10b49b6f5","Type":"ContainerStarted","Data":"3f857a92decce35c762072463303b72a975ae47dccf57f0364ccb565a6b151f3"}
Sep 16 18:45:02.744820 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.744769    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lrj6j" event={"ID":"1e6040a3-2386-4096-90e7-fb3bec2c2f60","Type":"ContainerStarted","Data":"58cdcb4f694688b62d18d277388e0dd4afb2650be74bd176bde029b36fb4ca26"}
Sep 16 18:45:02.746340 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.746317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4vw5f" event={"ID":"1599ebeb-4855-44c5-848f-b381ca2fb434","Type":"ContainerStarted","Data":"caaab697507d9cc5c7e5a0375c488b7229a2e900a5cd813f3eb729389ccdf648"}
Sep 16 18:45:02.747548 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:02.747525    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" event={"ID":"c7de017d6d245b795fdd5ece3d22237c","Type":"ContainerStarted","Data":"3fe3f95e24d30c18dba393be3243274bf188d4b90266410263006baaa8499df7"}
Sep 16 18:45:02.781793 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:02.781757    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3087807d_e287_4a6d_b99c_17aa3ab3e6a5.slice/crio-9c6054c06c1af0e42bd36928b5518963d370db01ddc017f95bc9983c52d72f2e WatchSource:0}: Error finding container 9c6054c06c1af0e42bd36928b5518963d370db01ddc017f95bc9983c52d72f2e: Status 404 returned error can't find the container with id 9c6054c06c1af0e42bd36928b5518963d370db01ddc017f95bc9983c52d72f2e
Sep 16 18:45:02.823759 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:02.823723    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca39171_efab_4995_a0fb_2261a074d3c9.slice/crio-84e5120c2121b6122d3f6523874cbaa685242d76fcfcba4403a2da755fca2e9c WatchSource:0}: Error finding container 84e5120c2121b6122d3f6523874cbaa685242d76fcfcba4403a2da755fca2e9c: Status 404 returned error can't find the container with id 84e5120c2121b6122d3f6523874cbaa685242d76fcfcba4403a2da755fca2e9c
Sep 16 18:45:03.216389 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:03.216344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:03.216595 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:03.216564    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:03.216672 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:03.216632    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret podName:0c7a47e0-a2e8-4c1d-815b-18c9cecea99d nodeName:}" failed. No retries permitted until 2026-09-16 18:45:05.216613052 +0000 UTC m=+5.019331254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret") pod "global-pull-secret-syncer-l6stq" (UID: "0c7a47e0-a2e8-4c1d-815b-18c9cecea99d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:03.318533 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:03.317222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs\") pod \"network-metrics-daemon-65n74\" (UID: \"6b8c5f93-a582-446f-873c-5b3344f1c2c5\") " pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:03.318533 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:03.317397    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:03.318533 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:03.317471    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs podName:6b8c5f93-a582-446f-873c-5b3344f1c2c5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:05.317448355 +0000 UTC m=+5.120166575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs") pod "network-metrics-daemon-65n74" (UID: "6b8c5f93-a582-446f-873c-5b3344f1c2c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:03.418668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:03.418624    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slf4b\" (UniqueName: \"kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b\") pod \"network-check-target-vmv47\" (UID: \"f247b77e-176f-4657-9447-ef0b89f708b3\") " pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:03.418892 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:03.418806    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:03.418892 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:03.418827    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:03.418892 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:03.418841    2561 projected.go:196] Error preparing data for projected volume kube-api-access-slf4b for pod openshift-network-diagnostics/network-check-target-vmv47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:03.419063 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:03.418908    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b podName:f247b77e-176f-4657-9447-ef0b89f708b3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:05.418885908 +0000 UTC m=+5.221604107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-slf4b" (UniqueName: "kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b") pod "network-check-target-vmv47" (UID: "f247b77e-176f-4657-9447-ef0b89f708b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:03.473420 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:03.473326    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:03.654179 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:03.654108    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:02 +0000 UTC" deadline="2028-02-17 15:41:15.719996574 +0000 UTC"
Sep 16 18:45:03.654179 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:03.654176    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12452h56m12.065824773s"
Sep 16 18:45:03.728617 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:03.728528    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:03.728799 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:03.728688    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:03.729194 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:03.729156    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:03.729319 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:03.729274    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:03.729398 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:03.729351    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:03.729456 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:03.729427    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:03.762587 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:03.762547    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hm84r" event={"ID":"0ca39171-efab-4995-a0fb-2261a074d3c9","Type":"ContainerStarted","Data":"84e5120c2121b6122d3f6523874cbaa685242d76fcfcba4403a2da755fca2e9c"}
Sep 16 18:45:03.772335 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:03.772296    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lvn8l" event={"ID":"3087807d-e287-4a6d-b99c-17aa3ab3e6a5","Type":"ContainerStarted","Data":"9c6054c06c1af0e42bd36928b5518963d370db01ddc017f95bc9983c52d72f2e"}
Sep 16 18:45:05.235435 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:05.235387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:05.236079 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:05.235573    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:05.236079 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:05.235645    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret podName:0c7a47e0-a2e8-4c1d-815b-18c9cecea99d nodeName:}" failed. No retries permitted until 2026-09-16 18:45:09.235625331 +0000 UTC m=+9.038343542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret") pod "global-pull-secret-syncer-l6stq" (UID: "0c7a47e0-a2e8-4c1d-815b-18c9cecea99d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:05.336575 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:05.336531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs\") pod \"network-metrics-daemon-65n74\" (UID: \"6b8c5f93-a582-446f-873c-5b3344f1c2c5\") " pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:05.336786 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:05.336714    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:05.336786 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:05.336786    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs podName:6b8c5f93-a582-446f-873c-5b3344f1c2c5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:09.336764158 +0000 UTC m=+9.139482371 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs") pod "network-metrics-daemon-65n74" (UID: "6b8c5f93-a582-446f-873c-5b3344f1c2c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:05.438209 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:05.437489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slf4b\" (UniqueName: \"kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b\") pod \"network-check-target-vmv47\" (UID: \"f247b77e-176f-4657-9447-ef0b89f708b3\") " pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:05.438209 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:05.437723    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:05.438209 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:05.437745    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:05.438209 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:05.437759    2561 projected.go:196] Error preparing data for projected volume kube-api-access-slf4b for pod openshift-network-diagnostics/network-check-target-vmv47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:05.438209 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:05.437819    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b podName:f247b77e-176f-4657-9447-ef0b89f708b3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:09.437801653 +0000 UTC m=+9.240519853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-slf4b" (UniqueName: "kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b") pod "network-check-target-vmv47" (UID: "f247b77e-176f-4657-9447-ef0b89f708b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:05.729360 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:05.728423    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:05.729360 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:05.728573    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:05.729360 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:05.729028    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:05.729360 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:05.729193    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:05.729360 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:05.729245    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:05.729360 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:05.729317    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:07.729166 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:07.728242    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:07.729166 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:07.728418    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:07.729166 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:07.728857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:07.729166 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:07.728949    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:07.729166 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:07.729012    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:07.729166 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:07.729095    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:09.277872 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:09.277829    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:09.278349 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:09.278030    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:09.278349 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:09.278125    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret podName:0c7a47e0-a2e8-4c1d-815b-18c9cecea99d nodeName:}" failed. No retries permitted until 2026-09-16 18:45:17.278081436 +0000 UTC m=+17.080799649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret") pod "global-pull-secret-syncer-l6stq" (UID: "0c7a47e0-a2e8-4c1d-815b-18c9cecea99d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:09.378625 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:09.378575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs\") pod \"network-metrics-daemon-65n74\" (UID: \"6b8c5f93-a582-446f-873c-5b3344f1c2c5\") " pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:09.378832 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:09.378734    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:09.378832 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:09.378811    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs podName:6b8c5f93-a582-446f-873c-5b3344f1c2c5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:17.37879218 +0000 UTC m=+17.181510376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs") pod "network-metrics-daemon-65n74" (UID: "6b8c5f93-a582-446f-873c-5b3344f1c2c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:09.479010 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:09.478964    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slf4b\" (UniqueName: \"kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b\") pod \"network-check-target-vmv47\" (UID: \"f247b77e-176f-4657-9447-ef0b89f708b3\") " pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:09.479230 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:09.479199    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:09.479230 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:09.479220    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:09.479343 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:09.479234    2561 projected.go:196] Error preparing data for projected volume kube-api-access-slf4b for pod openshift-network-diagnostics/network-check-target-vmv47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:09.479343 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:09.479288    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b podName:f247b77e-176f-4657-9447-ef0b89f708b3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:17.479274394 +0000 UTC m=+17.281992589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-slf4b" (UniqueName: "kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b") pod "network-check-target-vmv47" (UID: "f247b77e-176f-4657-9447-ef0b89f708b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:09.728396 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:09.728347    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:09.728609 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:09.728548    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:09.728993 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:09.728973    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:09.729094 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:09.729064    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:09.729157 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:09.729119    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:09.729268 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:09.729211    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:11.728546 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:11.727999    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:11.728546 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:11.728083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:11.728546 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:11.728168    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:11.728546 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:11.728244    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:11.728546 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:11.728346    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:11.728546 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:11.728438    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:13.728925 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:13.728884    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:13.728925 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:13.728915    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:13.728925 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:13.728924    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:13.729436 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:13.729021    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:13.729436 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:13.729103    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:13.729436 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:13.729136    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:15.728074 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:15.728017    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:15.728074 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:15.728047    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:15.728542 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:15.728180    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:15.728542 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:15.728207    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:15.728542 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:15.728244    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:15.728542 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:15.728311    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:17.339175 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:17.339131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:17.339597 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:17.339302    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:17.339597 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:17.339395    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret podName:0c7a47e0-a2e8-4c1d-815b-18c9cecea99d nodeName:}" failed. No retries permitted until 2026-09-16 18:45:33.339372604 +0000 UTC m=+33.142090840 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret") pod "global-pull-secret-syncer-l6stq" (UID: "0c7a47e0-a2e8-4c1d-815b-18c9cecea99d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:17.439879 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:17.439834    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs\") pod \"network-metrics-daemon-65n74\" (UID: \"6b8c5f93-a582-446f-873c-5b3344f1c2c5\") " pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:17.440096 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:17.439984    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:17.440096 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:17.440072    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs podName:6b8c5f93-a582-446f-873c-5b3344f1c2c5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:33.440036002 +0000 UTC m=+33.242754201 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs") pod "network-metrics-daemon-65n74" (UID: "6b8c5f93-a582-446f-873c-5b3344f1c2c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:17.541073 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:17.541013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slf4b\" (UniqueName: \"kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b\") pod \"network-check-target-vmv47\" (UID: \"f247b77e-176f-4657-9447-ef0b89f708b3\") " pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:17.541251 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:17.541199    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:17.541251 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:17.541230    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:17.541251 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:17.541247    2561 projected.go:196] Error preparing data for projected volume kube-api-access-slf4b for pod openshift-network-diagnostics/network-check-target-vmv47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:17.541379 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:17.541318    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b podName:f247b77e-176f-4657-9447-ef0b89f708b3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:33.541299443 +0000 UTC m=+33.344017638 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-slf4b" (UniqueName: "kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b") pod "network-check-target-vmv47" (UID: "f247b77e-176f-4657-9447-ef0b89f708b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:17.728513 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:17.728470    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:17.728513 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:17.728502    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:17.728763 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:17.728485    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:17.728763 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:17.728610    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:17.728763 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:17.728692    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:17.728763 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:17.728751    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:19.728225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:19.728170    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:19.728225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:19.728194    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:19.728674 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:19.728194    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:19.728674 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:19.728313    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:19.728674 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:19.728411    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:19.728674 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:19.728489    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:21.727959 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:21.727913    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:21.728414 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:21.727914    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:21.728414 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:21.728042    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:21.728414 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:21.727927    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:21.728414 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:21.728155    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:21.728414 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:21.728225    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:23.728171 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.727946    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:23.728835 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.727951    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:23.728835 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:23.728256    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:23.728835 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.727963    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:23.728835 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:23.728312    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:23.728835 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:23.728376    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:23.824168 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.824125    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ldqrg" event={"ID":"a5223494-bd2b-4911-ae88-8bb10b49b6f5","Type":"ContainerStarted","Data":"79c82491ef637f16142047fd2806185dd77a6f25b5d0f4db83930770a9499854"}
Sep 16 18:45:23.825794 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.825768    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lrj6j" event={"ID":"1e6040a3-2386-4096-90e7-fb3bec2c2f60","Type":"ContainerStarted","Data":"f885c4283be2e0d6e1a6f5dae1b1cdbb61eba0bd64ebc084028060f3d1b88c11"}
Sep 16 18:45:23.827253 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.827220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4vw5f" event={"ID":"1599ebeb-4855-44c5-848f-b381ca2fb434","Type":"ContainerStarted","Data":"24dc305f77772b443711734329024e73287e9f303aa8c6902f8041f115f317ca"}
Sep 16 18:45:23.828743 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.828723    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" event={"ID":"c7de017d6d245b795fdd5ece3d22237c","Type":"ContainerStarted","Data":"d63ba19b772a9caeb1288f485dab3c7936202dc0bcb89d6c13273743479a91eb"}
Sep 16 18:45:23.831380 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.831361    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 18:45:23.831677 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.831659    2561 generic.go:356] "Generic (PLEG): container finished" podID="5e84101c-123e-475f-ad77-fbe61af69584" containerID="2bcbbcb42d277404aacfb5b3fd42b827c297536078fe2bbc5d8c08c7375d6407" exitCode=1
Sep 16 18:45:23.831752 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.831731    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7" event={"ID":"5e84101c-123e-475f-ad77-fbe61af69584","Type":"ContainerStarted","Data":"24c8bbdd02d2df8254003a18949a594c9c4747a486f3894307983ac45bf7046a"}
Sep 16 18:45:23.831826 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.831810    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7" event={"ID":"5e84101c-123e-475f-ad77-fbe61af69584","Type":"ContainerStarted","Data":"c72069ffc781a4592eafd93a63f6c53aad0f8cec96943b5ebe0fc092e38d05af"}
Sep 16 18:45:23.831880 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.831868    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7" event={"ID":"5e84101c-123e-475f-ad77-fbe61af69584","Type":"ContainerStarted","Data":"a37c52f4c5d53b634814b35d2a9bc75d42e97e405748131fe64da8a6aaef3088"}
Sep 16 18:45:23.831931 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.831918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7" event={"ID":"5e84101c-123e-475f-ad77-fbe61af69584","Type":"ContainerStarted","Data":"713dd93178e0f2639d55d0437e57014bfcce795b9d1171ea70c1275059a225a5"}
Sep 16 18:45:23.831974 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.831957    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7" event={"ID":"5e84101c-123e-475f-ad77-fbe61af69584","Type":"ContainerDied","Data":"2bcbbcb42d277404aacfb5b3fd42b827c297536078fe2bbc5d8c08c7375d6407"}
Sep 16 18:45:23.832018 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.831987    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7" event={"ID":"5e84101c-123e-475f-ad77-fbe61af69584","Type":"ContainerStarted","Data":"d8872989ec9a5e2524e077a9f3fd4cd68ea1171e8de4c2dd1046311ae20660eb"}
Sep 16 18:45:23.833202 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.833179    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ca4a1da-ce89-4def-9f1f-01f4ea36b08d" containerID="cba7de4b9dfb84f8f1b84f84ee69fe9d94ba772a39c519d460b467064894b031" exitCode=0
Sep 16 18:45:23.833292 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.833248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ck2ml" event={"ID":"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d","Type":"ContainerDied","Data":"cba7de4b9dfb84f8f1b84f84ee69fe9d94ba772a39c519d460b467064894b031"}
Sep 16 18:45:23.834853 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.834836    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw" event={"ID":"7d21675b-89e7-4f54-8340-6228f09668e9","Type":"ContainerStarted","Data":"6054e97f1dd17e3621da58dffd40aa954de4bbd0ef58b6ea824b9b5a3d16d017"}
Sep 16 18:45:23.836422 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.836401    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d69cc83b62a0c0df434d27c89a80e90" containerID="0e2bc0e117c02f0fa9acc60eacb2c7a75f0a9637a575cb94b30e785b27e5fda1" exitCode=0
Sep 16 18:45:23.836514 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.836486    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" event={"ID":"3d69cc83b62a0c0df434d27c89a80e90","Type":"ContainerDied","Data":"0e2bc0e117c02f0fa9acc60eacb2c7a75f0a9637a575cb94b30e785b27e5fda1"}
Sep 16 18:45:23.837886 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.837864    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hm84r" event={"ID":"0ca39171-efab-4995-a0fb-2261a074d3c9","Type":"ContainerStarted","Data":"31ab1f07231e5212a9e6bc30071a39e1e71821c2fb378326dfe7a5d96ef9482a"}
Sep 16 18:45:23.839292 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.839273    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lvn8l" event={"ID":"3087807d-e287-4a6d-b99c-17aa3ab3e6a5","Type":"ContainerStarted","Data":"713fd48516acca4fe160013ee561abd2fd386076562a1ae98a8d54ee76c5d759"}
Sep 16 18:45:23.840264 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.840217    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ldqrg" podStartSLOduration=2.666670733 podStartE2EDuration="22.840205883s" totalImagesPullingTime="20.17353515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:23.840181042 +0000 UTC m=+23.642899258" watchObservedRunningTime="2026-09-16 18:45:23.840205883 +0000 UTC m=+23.642924108"
Sep 16 18:45:23.859315 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.859271    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" podStartSLOduration=21.859257164 podStartE2EDuration="21.859257164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:23.858846994 +0000 UTC m=+23.661565221" watchObservedRunningTime="2026-09-16 18:45:23.859257164 +0000 UTC m=+23.661975376"
Sep 16 18:45:23.879495 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.879443    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4vw5f" podStartSLOduration=1.77672605 podStartE2EDuration="22.879421893s" totalImagesPullingTime="21.102695843s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:23.87909911 +0000 UTC m=+23.681817327" watchObservedRunningTime="2026-09-16 18:45:23.879421893 +0000 UTC m=+23.682140110"
Sep 16 18:45:23.899530 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.899437    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lrj6j" podStartSLOduration=2.661790318 podStartE2EDuration="22.899417903s" totalImagesPullingTime="20.237627585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:23.899035664 +0000 UTC m=+23.701753904" watchObservedRunningTime="2026-09-16 18:45:23.899417903 +0000 UTC m=+23.702136120"
Sep 16 18:45:23.916273 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.916228    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hm84r" podStartSLOduration=3.002909862 podStartE2EDuration="22.916215499s" totalImagesPullingTime="19.913305637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:23.916197735 +0000 UTC m=+23.718915950" watchObservedRunningTime="2026-09-16 18:45:23.916215499 +0000 UTC m=+23.718933715"
Sep 16 18:45:23.986450 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:23.986173    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lvn8l" podStartSLOduration=3.036980656 podStartE2EDuration="22.986151527s" totalImagesPullingTime="19.949170871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:23.962156386 +0000 UTC m=+23.764874595" watchObservedRunningTime="2026-09-16 18:45:23.986151527 +0000 UTC m=+23.788869745"
Sep 16 18:45:24.148439 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:24.148397    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:24.231564 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:24.231524    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:24.844938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:24.844537    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q79dm" event={"ID":"7318780a-dbbe-4905-9e89-ce48d61c0d30","Type":"ContainerStarted","Data":"82658fc5b47b2655a3aa199c75e4c2be12c83b58e1b21a8ac46b1bd9c8562f37"}
Sep 16 18:45:24.846921 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:24.846872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" event={"ID":"3d69cc83b62a0c0df434d27c89a80e90","Type":"ContainerStarted","Data":"d252626f0be90ce9afe4475f67990ed5294095b52c8512f5b9c56738e9ecb5c5"}
Sep 16 18:45:24.860810 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:24.860769    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q79dm" podStartSLOduration=3.805461294 podStartE2EDuration="23.860749715s" totalImagesPullingTime="20.055288421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:24.860461539 +0000 UTC m=+24.663179758" watchObservedRunningTime="2026-09-16 18:45:24.860749715 +0000 UTC m=+24.663467955"
Sep 16 18:45:24.877757 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:24.877703    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" podStartSLOduration=22.877682615 podStartE2EDuration="22.877682615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:24.87703028 +0000 UTC m=+24.679748493" watchObservedRunningTime="2026-09-16 18:45:24.877682615 +0000 UTC m=+24.680400833"
Sep 16 18:45:24.901815 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:24.901775    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:45:24.984400 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:24.984359    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:25.072588 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:25.072503    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:25.674298 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:25.674183    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:45:24.901794946Z","UUID":"8c0df4bd-eda8-4481-b4f9-ed08eb0ff1a9","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:45:25.676601 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:25.676577    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:45:25.676729 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:25.676619    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:45:25.728566 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:25.728538    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:25.728729 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:25.728650    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:25.728729 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:25.728725    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:25.728831 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:25.728809    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:25.728895 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:25.728870    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:25.728957 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:25.728933    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:25.857155 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:25.857077    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 18:45:25.857630 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:25.857600    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7" event={"ID":"5e84101c-123e-475f-ad77-fbe61af69584","Type":"ContainerStarted","Data":"5f25f24bf5a1b0f70dc04cc32aa29eea656c1f55619993887a4a631dc9b6c8ef"}
Sep 16 18:45:25.860805 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:25.860780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw" event={"ID":"7d21675b-89e7-4f54-8340-6228f09668e9","Type":"ContainerStarted","Data":"e6c5233775ce1f5763d0a6499754f7813e650f655dc960e7653c99940d67d1d5"}
Sep 16 18:45:27.590881 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:27.590842    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lvn8l"
Sep 16 18:45:27.591660 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:27.591640    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lvn8l"
Sep 16 18:45:27.728217 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:27.728172    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:27.728217 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:27.728221    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:27.728461 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:27.728180    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:27.728461 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:27.728304    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:27.728461 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:27.728406    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:27.728584 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:27.728487    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:27.865139 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:27.865044    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lvn8l"
Sep 16 18:45:27.865599 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:27.865577    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lvn8l"
Sep 16 18:45:27.938329 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:27.938293    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:28.481814 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:28.481641    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:28.582176 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:28.582145    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:28.868656 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:28.868624    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ca4a1da-ce89-4def-9f1f-01f4ea36b08d" containerID="4f7c399abf2e8b2b1f37c0ccb853671ec74ed962701b97de2730d9df5088fb4f" exitCode=0
Sep 16 18:45:28.869255 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:28.868704    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ck2ml" event={"ID":"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d","Type":"ContainerDied","Data":"4f7c399abf2e8b2b1f37c0ccb853671ec74ed962701b97de2730d9df5088fb4f"}
Sep 16 18:45:28.871629 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:28.871564    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw" event={"ID":"7d21675b-89e7-4f54-8340-6228f09668e9","Type":"ContainerStarted","Data":"6cb3231a30316b1082f45f56fb6e73493e73b95005ccac5a4ce3e90eb8ce5cf0"}
Sep 16 18:45:28.879743 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:28.879722    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 18:45:28.880110 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:28.880082    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7" event={"ID":"5e84101c-123e-475f-ad77-fbe61af69584","Type":"ContainerStarted","Data":"fa2e4a396ecd71f3faac8229c57ec5c30f9aa835bba91170d9e38dedf7aa1260"}
Sep 16 18:45:28.880523 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:28.880506    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:28.880581 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:28.880572    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:28.880622 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:28.880610    2561 scope.go:117] "RemoveContainer" containerID="2bcbbcb42d277404aacfb5b3fd42b827c297536078fe2bbc5d8c08c7375d6407"
Sep 16 18:45:28.897847 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:28.897824    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:28.919512 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:28.919463    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkhpw" podStartSLOduration=3.590415106 podStartE2EDuration="27.919446109s" totalImagesPullingTime="24.329031003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:28.918858725 +0000 UTC m=+28.721576941" watchObservedRunningTime="2026-09-16 18:45:28.919446109 +0000 UTC m=+28.722164362"
Sep 16 18:45:29.728608 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:29.728567    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:29.728608 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:29.728593    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:29.728862 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:29.728590    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:29.728862 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:29.728722    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:29.728862 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:29.728761    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:29.728862 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:29.728832    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:29.890633 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:29.890426    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 18:45:29.892998 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:29.892963    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7" event={"ID":"5e84101c-123e-475f-ad77-fbe61af69584","Type":"ContainerStarted","Data":"de9169519e709d1159b689c5ac340f688c67d2323a0860455b76b26164ed3291"}
Sep 16 18:45:29.893534 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:29.893510    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:29.911966 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:29.911939    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:45:29.922138 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:29.922083    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7" podStartSLOduration=8.778675317 podStartE2EDuration="28.922046307s" totalImagesPullingTime="20.14337099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:29.921947574 +0000 UTC m=+29.724665789" watchObservedRunningTime="2026-09-16 18:45:29.922046307 +0000 UTC m=+29.724764523"
Sep 16 18:45:30.069389 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:30.069281    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l6stq"]
Sep 16 18:45:30.069531 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:30.069424    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:30.069531 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:30.069500    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:30.075706 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:30.075621    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-65n74"]
Sep 16 18:45:30.075860 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:30.075821    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:30.075997 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:30.075970    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:30.076294 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:30.076272    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vmv47"]
Sep 16 18:45:30.076386 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:30.076373    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:30.076485 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:30.076467    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:30.895522 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:30.895485    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ca4a1da-ce89-4def-9f1f-01f4ea36b08d" containerID="bf20cdec9bb45e6520baaf1d3d07761493415cc94134a67000c709357935b6aa" exitCode=0
Sep 16 18:45:30.895891 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:30.895559    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ck2ml" event={"ID":"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d","Type":"ContainerDied","Data":"bf20cdec9bb45e6520baaf1d3d07761493415cc94134a67000c709357935b6aa"}
Sep 16 18:45:31.465725 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:31.465670    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:31.728874 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:31.728792    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:31.729008 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:31.728792    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:31.729008 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:31.728891    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:31.729008 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:31.728792    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:31.729008 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:31.728976    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:31.729176 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:31.729037    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:31.900643 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:31.900612    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ca4a1da-ce89-4def-9f1f-01f4ea36b08d" containerID="f3236ee66e62d8a967d423d1b54ce2cf47dd3eaf9dc05bdb18fb961b0d0590dc" exitCode=0
Sep 16 18:45:31.901164 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:31.900684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ck2ml" event={"ID":"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d","Type":"ContainerDied","Data":"f3236ee66e62d8a967d423d1b54ce2cf47dd3eaf9dc05bdb18fb961b0d0590dc"}
Sep 16 18:45:33.224117 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:33.224084    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:33.358775 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:33.358721    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:33.358941 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:33.358895    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:33.358997 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:33.358980    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret podName:0c7a47e0-a2e8-4c1d-815b-18c9cecea99d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:05.358956346 +0000 UTC m=+65.161674540 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret") pod "global-pull-secret-syncer-l6stq" (UID: "0c7a47e0-a2e8-4c1d-815b-18c9cecea99d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:33.459530 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:33.459481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs\") pod \"network-metrics-daemon-65n74\" (UID: \"6b8c5f93-a582-446f-873c-5b3344f1c2c5\") " pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:33.459710 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:33.459630    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:33.459710 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:33.459710    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs podName:6b8c5f93-a582-446f-873c-5b3344f1c2c5 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:05.459690498 +0000 UTC m=+65.262408695 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs") pod "network-metrics-daemon-65n74" (UID: "6b8c5f93-a582-446f-873c-5b3344f1c2c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:33.470689 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:33.470643    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:33.560035 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:33.560006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slf4b\" (UniqueName: \"kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b\") pod \"network-check-target-vmv47\" (UID: \"f247b77e-176f-4657-9447-ef0b89f708b3\") " pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:33.560226 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:33.560173    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:33.560226 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:33.560191    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:33.560226 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:33.560200    2561 projected.go:196] Error preparing data for projected volume kube-api-access-slf4b for pod openshift-network-diagnostics/network-check-target-vmv47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:33.560357 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:33.560257    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b podName:f247b77e-176f-4657-9447-ef0b89f708b3 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:05.560239732 +0000 UTC m=+65.362957926 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-slf4b" (UniqueName: "kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b") pod "network-check-target-vmv47" (UID: "f247b77e-176f-4657-9447-ef0b89f708b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:33.573449 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:33.573412    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:33.728160 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:33.728126    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:33.728160 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:33.728127    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:33.728397 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:33.728145    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:33.728397 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:33.728265    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65n74" podUID="6b8c5f93-a582-446f-873c-5b3344f1c2c5"
Sep 16 18:45:33.728397 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:33.728374    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6stq" podUID="0c7a47e0-a2e8-4c1d-815b-18c9cecea99d"
Sep 16 18:45:33.728541 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:33.728470    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmv47" podUID="f247b77e-176f-4657-9447-ef0b89f708b3"
Sep 16 18:45:34.972424 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:34.972391    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeReady"
Sep 16 18:45:34.972912 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:34.972549    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:45:35.021607 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.021573    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dk7qf"]
Sep 16 18:45:35.025714 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.025689    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:35.026228 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.026203    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kthn2"]
Sep 16 18:45:35.028825 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.028800    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kthn2"
Sep 16 18:45:35.028934 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.028876    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:45:35.029452 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.029433    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dwfv6\""
Sep 16 18:45:35.029859 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.029841    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:45:35.031820 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.031799    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:45:35.032110 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.032084    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:45:35.032110 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.032106    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:45:35.032268 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.032106    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dgnzh\""
Sep 16 18:45:35.037207 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.037182    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dk7qf"]
Sep 16 18:45:35.038927 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.038897    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kthn2"]
Sep 16 18:45:35.165893 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.165814    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:35.170219 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.170182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ce5cd3-f535-4b56-8a33-e47719901e55-tmp-dir\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:35.170371 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.170232    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:35.170371 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.170283    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ce5cd3-f535-4b56-8a33-e47719901e55-config-volume\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:35.170487 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.170377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtt2m\" (UniqueName: \"kubernetes.io/projected/f6ce5cd3-f535-4b56-8a33-e47719901e55-kube-api-access-jtt2m\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:35.170487 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.170408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lsrm\" (UniqueName: \"kubernetes.io/projected/ae49ada0-3615-4001-8c5d-4fa78bbda02d-kube-api-access-5lsrm\") pod \"ingress-canary-kthn2\" (UID: \"ae49ada0-3615-4001-8c5d-4fa78bbda02d\") " pod="openshift-ingress-canary/ingress-canary-kthn2"
Sep 16 18:45:35.170487 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.170449    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae49ada0-3615-4001-8c5d-4fa78bbda02d-cert\") pod \"ingress-canary-kthn2\" (UID: \"ae49ada0-3615-4001-8c5d-4fa78bbda02d\") " pod="openshift-ingress-canary/ingress-canary-kthn2"
Sep 16 18:45:35.228149 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.228106    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:35.271094 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.271027    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ce5cd3-f535-4b56-8a33-e47719901e55-tmp-dir\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:35.271276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.271121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:35.271276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.271148    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ce5cd3-f535-4b56-8a33-e47719901e55-config-volume\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:35.271276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.271208    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtt2m\" (UniqueName: \"kubernetes.io/projected/f6ce5cd3-f535-4b56-8a33-e47719901e55-kube-api-access-jtt2m\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:35.271276 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.271237    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lsrm\" (UniqueName: \"kubernetes.io/projected/ae49ada0-3615-4001-8c5d-4fa78bbda02d-kube-api-access-5lsrm\") pod \"ingress-canary-kthn2\" (UID: \"ae49ada0-3615-4001-8c5d-4fa78bbda02d\") " pod="openshift-ingress-canary/ingress-canary-kthn2"
Sep 16 18:45:35.271473 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.271278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae49ada0-3615-4001-8c5d-4fa78bbda02d-cert\") pod \"ingress-canary-kthn2\" (UID: \"ae49ada0-3615-4001-8c5d-4fa78bbda02d\") " pod="openshift-ingress-canary/ingress-canary-kthn2"
Sep 16 18:45:35.271473 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:35.271309    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:35.271473 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.271392    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ce5cd3-f535-4b56-8a33-e47719901e55-tmp-dir\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:35.271473 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:35.271399    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls podName:f6ce5cd3-f535-4b56-8a33-e47719901e55 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.771375663 +0000 UTC m=+35.574093871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls") pod "dns-default-dk7qf" (UID: "f6ce5cd3-f535-4b56-8a33-e47719901e55") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:35.272028 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.271997    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ce5cd3-f535-4b56-8a33-e47719901e55-config-volume\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:35.276635 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.276610    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae49ada0-3615-4001-8c5d-4fa78bbda02d-cert\") pod \"ingress-canary-kthn2\" (UID: \"ae49ada0-3615-4001-8c5d-4fa78bbda02d\") " pod="openshift-ingress-canary/ingress-canary-kthn2"
Sep 16 18:45:35.280904 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.280879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtt2m\" (UniqueName: \"kubernetes.io/projected/f6ce5cd3-f535-4b56-8a33-e47719901e55-kube-api-access-jtt2m\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:35.281179 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.281153    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lsrm\" (UniqueName: \"kubernetes.io/projected/ae49ada0-3615-4001-8c5d-4fa78bbda02d-kube-api-access-5lsrm\") pod \"ingress-canary-kthn2\" (UID: \"ae49ada0-3615-4001-8c5d-4fa78bbda02d\") " pod="openshift-ingress-canary/ingress-canary-kthn2"
Sep 16 18:45:35.346114 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.346076    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kthn2"
Sep 16 18:45:35.727891 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.727854    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:45:35.728084 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.727854    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:45:35.728084 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.727854    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:45:35.732143 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.732122    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:35.732143 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.732137    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:35.732327 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.732167    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:35.732327 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.732137    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pq9cp\""
Sep 16 18:45:35.732327 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.732137    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:35.732536 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.732521    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hrh8k\""
Sep 16 18:45:35.776556 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:35.776524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:35.776720 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:35.776691    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:35.776785 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:35.776773    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls podName:f6ce5cd3-f535-4b56-8a33-e47719901e55 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:36.776755764 +0000 UTC m=+36.579473957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls") pod "dns-default-dk7qf" (UID: "f6ce5cd3-f535-4b56-8a33-e47719901e55") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:36.091664 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:36.091085    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kthn2"]
Sep 16 18:45:36.104657 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:45:36.104631    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae49ada0_3615_4001_8c5d_4fa78bbda02d.slice/crio-4543a602751e8e39cd86b6f6f7a76d39e85c50716149843d1b066b1a775dc115 WatchSource:0}: Error finding container 4543a602751e8e39cd86b6f6f7a76d39e85c50716149843d1b066b1a775dc115: Status 404 returned error can't find the container with id 4543a602751e8e39cd86b6f6f7a76d39e85c50716149843d1b066b1a775dc115
Sep 16 18:45:36.783338 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:36.783299    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:36.783551 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:36.783439    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:36.783551 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:36.783515    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls podName:f6ce5cd3-f535-4b56-8a33-e47719901e55 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:38.783493261 +0000 UTC m=+38.586211455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls") pod "dns-default-dk7qf" (UID: "f6ce5cd3-f535-4b56-8a33-e47719901e55") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:36.917737 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:36.917688    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ca4a1da-ce89-4def-9f1f-01f4ea36b08d" containerID="ab644746216817263f2df5b370fd0d42a4787987e04cde5903b812855d4cbe19" exitCode=0
Sep 16 18:45:36.917914 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:36.917742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ck2ml" event={"ID":"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d","Type":"ContainerDied","Data":"ab644746216817263f2df5b370fd0d42a4787987e04cde5903b812855d4cbe19"}
Sep 16 18:45:36.919224 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:36.919193    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kthn2" event={"ID":"ae49ada0-3615-4001-8c5d-4fa78bbda02d","Type":"ContainerStarted","Data":"4543a602751e8e39cd86b6f6f7a76d39e85c50716149843d1b066b1a775dc115"}
Sep 16 18:45:37.925520 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:37.925482    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ca4a1da-ce89-4def-9f1f-01f4ea36b08d" containerID="0c7a5236fb19f0f83639c31d2cd107a21868a8a57d96370f12cc6a746f704736" exitCode=0
Sep 16 18:45:37.926146 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:37.925523    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ck2ml" event={"ID":"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d","Type":"ContainerDied","Data":"0c7a5236fb19f0f83639c31d2cd107a21868a8a57d96370f12cc6a746f704736"}
Sep 16 18:45:38.801200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:38.800949    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:38.801346 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:38.801096    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:38.801346 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:38.801317    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls podName:f6ce5cd3-f535-4b56-8a33-e47719901e55 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:42.801300831 +0000 UTC m=+42.604019025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls") pod "dns-default-dk7qf" (UID: "f6ce5cd3-f535-4b56-8a33-e47719901e55") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:38.930034 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:38.929919    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kthn2" event={"ID":"ae49ada0-3615-4001-8c5d-4fa78bbda02d","Type":"ContainerStarted","Data":"a9c7dc269691cc437e5a8e1efc7db2bcd6c85f1c0106407dff83810cba0704d7"}
Sep 16 18:45:38.933551 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:38.933521    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ck2ml" event={"ID":"7ca4a1da-ce89-4def-9f1f-01f4ea36b08d","Type":"ContainerStarted","Data":"90fc04964caca9e5de4e4a6508463050818b13397738e4f081fc1a840b7282a3"}
Sep 16 18:45:38.950694 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:38.950627    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kthn2" podStartSLOduration=1.428883274 podStartE2EDuration="3.950596979s" totalImagesPullingTime="2.521713705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:38.950228311 +0000 UTC m=+38.752946528" watchObservedRunningTime="2026-09-16 18:45:38.950596979 +0000 UTC m=+38.753315215"
Sep 16 18:45:38.976222 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:38.976153    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ck2ml" podStartSLOduration=7.701177648 podStartE2EDuration="37.976136888s" totalImagesPullingTime="30.27495924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:38.974686712 +0000 UTC m=+38.777404928" watchObservedRunningTime="2026-09-16 18:45:38.976136888 +0000 UTC m=+38.778855104"
Sep 16 18:45:39.159484 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:39.159449    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:39.295325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:39.295297    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:41.370599 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:41.370567    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:41.471961 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:41.471928    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:41.631291 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:41.631202    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:42.830235 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:42.830192    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:42.830631 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:42.830354    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:42.830631 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:42.830425    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls podName:f6ce5cd3-f535-4b56-8a33-e47719901e55 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:50.830410117 +0000 UTC m=+50.633128311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls") pod "dns-default-dk7qf" (UID: "f6ce5cd3-f535-4b56-8a33-e47719901e55") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:50.335612 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:50.335573    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:50.894596 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:50.894561    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:45:50.894765 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:50.894718    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:50.894806 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:45:50.894786    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls podName:f6ce5cd3-f535-4b56-8a33-e47719901e55 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:06.894767186 +0000 UTC m=+66.697485384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls") pod "dns-default-dk7qf" (UID: "f6ce5cd3-f535-4b56-8a33-e47719901e55") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:56.518928 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:56.518862    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:56.587747 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:56.587707    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:56.662350 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:45:56.662323    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:01.912309 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:01.912280    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-chgz7"
Sep 16 18:46:02.056504 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:02.056472    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:02.136181 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:02.136141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:05.166637 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.166599    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz"]
Sep 16 18:46:05.194235 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.194195    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"]
Sep 16 18:46:05.194386 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.194328    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz"
Sep 16 18:46:05.197501 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.197470    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:05.197634 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.197528    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:05.197634 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.197570    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 18:46:05.197634 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.197628    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-fvnbh\""
Sep 16 18:46:05.197794 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.197753    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 18:46:05.208117 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.208096    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz"]
Sep 16 18:46:05.208222 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.208125    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"]
Sep 16 18:46:05.208222 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.208214    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"
Sep 16 18:46:05.213034 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.213012    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 18:46:05.213322 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.213016    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:05.213322 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.213022    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:05.213322 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.213107    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2hm9r\""
Sep 16 18:46:05.264964 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.264932    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"]
Sep 16 18:46:05.282179 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.282151    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-f2clx"]
Sep 16 18:46:05.282297 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.282279    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:05.285993 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.285902    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bg9x5\""
Sep 16 18:46:05.285993 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.285977    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 18:46:05.288179 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.288159    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 18:46:05.288349 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.288187    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:46:05.288475 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.288201    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:46:05.297329 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.297294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c177863-7189-4c95-8725-4c72172ffcf1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-p8frz\" (UID: \"9c177863-7189-4c95-8725-4c72172ffcf1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz"
Sep 16 18:46:05.297436 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.297350    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c177863-7189-4c95-8725-4c72172ffcf1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-p8frz\" (UID: \"9c177863-7189-4c95-8725-4c72172ffcf1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz"
Sep 16 18:46:05.297436 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.297370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9265\" (UniqueName: \"kubernetes.io/projected/9c177863-7189-4c95-8725-4c72172ffcf1-kube-api-access-n9265\") pod \"kube-storage-version-migrator-operator-6c8f84f897-p8frz\" (UID: \"9c177863-7189-4c95-8725-4c72172ffcf1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz"
Sep 16 18:46:05.308117 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.308093    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"]
Sep 16 18:46:05.308245 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.308142    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-f2clx"]
Sep 16 18:46:05.308245 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.308144    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.310815 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.310795    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 18:46:05.310936 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.310866    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:46:05.310936 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.310892    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bbx8j\""
Sep 16 18:46:05.310936 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.310909    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 18:46:05.311181 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.310976    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:46:05.316759 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.316741    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 18:46:05.354242 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.354211    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5884f9db8b-s8cjv"]
Sep 16 18:46:05.378284 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.378221    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5884f9db8b-s8cjv"]
Sep 16 18:46:05.378433 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.378332    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.381113 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.381084    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:46:05.381223 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.381178    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cnz8v\""
Sep 16 18:46:05.381223 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.381204    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:46:05.381309 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.381260    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:46:05.386410 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.386392    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:46:05.398006 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.397977    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c177863-7189-4c95-8725-4c72172ffcf1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-p8frz\" (UID: \"9c177863-7189-4c95-8725-4c72172ffcf1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz"
Sep 16 18:46:05.398157 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.398012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9265\" (UniqueName: \"kubernetes.io/projected/9c177863-7189-4c95-8725-4c72172ffcf1-kube-api-access-n9265\") pod \"kube-storage-version-migrator-operator-6c8f84f897-p8frz\" (UID: \"9c177863-7189-4c95-8725-4c72172ffcf1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz"
Sep 16 18:46:05.398157 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.398049    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:46:05.398157 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.398097    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-f7764\" (UID: \"08e67d03-f589-4064-862b-1fb49eff9bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:05.398157 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.398116    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/08e67d03-f589-4064-862b-1fb49eff9bb2-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-f7764\" (UID: \"08e67d03-f589-4064-862b-1fb49eff9bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:05.398157 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.398134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tm8zv\" (UID: \"90969688-8134-4f01-8002-b274aed1c2da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"
Sep 16 18:46:05.398405 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.398160    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fm6r\" (UniqueName: \"kubernetes.io/projected/08e67d03-f589-4064-862b-1fb49eff9bb2-kube-api-access-7fm6r\") pod \"cluster-monitoring-operator-6cd954b897-f7764\" (UID: \"08e67d03-f589-4064-862b-1fb49eff9bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:05.398405 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.398341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c177863-7189-4c95-8725-4c72172ffcf1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-p8frz\" (UID: \"9c177863-7189-4c95-8725-4c72172ffcf1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz"
Sep 16 18:46:05.398405 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.398363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwvqv\" (UniqueName: \"kubernetes.io/projected/90969688-8134-4f01-8002-b274aed1c2da-kube-api-access-rwvqv\") pod \"cluster-samples-operator-86499c89c4-tm8zv\" (UID: \"90969688-8134-4f01-8002-b274aed1c2da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"
Sep 16 18:46:05.398818 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.398797    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c177863-7189-4c95-8725-4c72172ffcf1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-p8frz\" (UID: \"9c177863-7189-4c95-8725-4c72172ffcf1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz"
Sep 16 18:46:05.400781 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.400764    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:46:05.401517 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.401488    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c177863-7189-4c95-8725-4c72172ffcf1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-p8frz\" (UID: \"9c177863-7189-4c95-8725-4c72172ffcf1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz"
Sep 16 18:46:05.407178 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.407146    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9265\" (UniqueName: \"kubernetes.io/projected/9c177863-7189-4c95-8725-4c72172ffcf1-kube-api-access-n9265\") pod \"kube-storage-version-migrator-operator-6c8f84f897-p8frz\" (UID: \"9c177863-7189-4c95-8725-4c72172ffcf1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz"
Sep 16 18:46:05.411020 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.410999    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c7a47e0-a2e8-4c1d-815b-18c9cecea99d-original-pull-secret\") pod \"global-pull-secret-syncer-l6stq\" (UID: \"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d\") " pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:46:05.443335 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.443251    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6stq"
Sep 16 18:46:05.454525 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.454495    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:05.498955 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.498918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tm8zv\" (UID: \"90969688-8134-4f01-8002-b274aed1c2da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"
Sep 16 18:46:05.498955 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.498964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.499225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.498996    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fm6r\" (UniqueName: \"kubernetes.io/projected/08e67d03-f589-4064-862b-1fb49eff9bb2-kube-api-access-7fm6r\") pod \"cluster-monitoring-operator-6cd954b897-f7764\" (UID: \"08e67d03-f589-4064-862b-1fb49eff9bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:05.499225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-certificates\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.499225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46hpx\" (UniqueName: \"kubernetes.io/projected/84e3975d-dbd7-4c94-a16c-e61da230a8c7-kube-api-access-46hpx\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.499225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs\") pod \"network-metrics-daemon-65n74\" (UID: \"6b8c5f93-a582-446f-873c-5b3344f1c2c5\") " pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:46:05.499225 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:05.499104    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:46:05.499225 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwvqv\" (UniqueName: \"kubernetes.io/projected/90969688-8134-4f01-8002-b274aed1c2da-kube-api-access-rwvqv\") pod \"cluster-samples-operator-86499c89c4-tm8zv\" (UID: \"90969688-8134-4f01-8002-b274aed1c2da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"
Sep 16 18:46:05.499225 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:05.499210    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls podName:90969688-8134-4f01-8002-b274aed1c2da nodeName:}" failed. No retries permitted until 2026-09-16 18:46:05.999185728 +0000 UTC m=+65.801903921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tm8zv" (UID: "90969688-8134-4f01-8002-b274aed1c2da") : secret "samples-operator-tls" not found
Sep 16 18:46:05.499568 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-image-registry-private-configuration\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.499568 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-trusted-ca\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.499568 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499300    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnngq\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-kube-api-access-nnngq\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.499568 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-installation-pull-secrets\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.499568 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499385    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-bound-sa-token\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.499568 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84e3975d-dbd7-4c94-a16c-e61da230a8c7-tmp\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.499568 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/84e3975d-dbd7-4c94-a16c-e61da230a8c7-snapshots\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.499568 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499483    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e3975d-dbd7-4c94-a16c-e61da230a8c7-trusted-ca-bundle\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.499568 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499525    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84e3975d-dbd7-4c94-a16c-e61da230a8c7-serving-cert\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.499903 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499600    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-f7764\" (UID: \"08e67d03-f589-4064-862b-1fb49eff9bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:05.499903 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499637    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/08e67d03-f589-4064-862b-1fb49eff9bb2-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-f7764\" (UID: \"08e67d03-f589-4064-862b-1fb49eff9bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:05.499903 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499666    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-ca-trust-extracted\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.499903 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.499702    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e3975d-dbd7-4c94-a16c-e61da230a8c7-service-ca-bundle\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.499903 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:05.499705    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:05.499903 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:05.499793    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls podName:08e67d03-f589-4064-862b-1fb49eff9bb2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:05.999778136 +0000 UTC m=+65.802496334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-f7764" (UID: "08e67d03-f589-4064-862b-1fb49eff9bb2") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:05.500259 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.500243    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/08e67d03-f589-4064-862b-1fb49eff9bb2-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-f7764\" (UID: \"08e67d03-f589-4064-862b-1fb49eff9bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:05.502898 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.502882    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz"
Sep 16 18:46:05.506177 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.506156    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:46:05.509314 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:05.509198    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:46:05.509314 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:05.509262    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs podName:6b8c5f93-a582-446f-873c-5b3344f1c2c5 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:09.509245106 +0000 UTC m=+129.311963306 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs") pod "network-metrics-daemon-65n74" (UID: "6b8c5f93-a582-446f-873c-5b3344f1c2c5") : secret "metrics-daemon-secret" not found
Sep 16 18:46:05.518898 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.518855    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwvqv\" (UniqueName: \"kubernetes.io/projected/90969688-8134-4f01-8002-b274aed1c2da-kube-api-access-rwvqv\") pod \"cluster-samples-operator-86499c89c4-tm8zv\" (UID: \"90969688-8134-4f01-8002-b274aed1c2da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"
Sep 16 18:46:05.519400 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.519349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fm6r\" (UniqueName: \"kubernetes.io/projected/08e67d03-f589-4064-862b-1fb49eff9bb2-kube-api-access-7fm6r\") pod \"cluster-monitoring-operator-6cd954b897-f7764\" (UID: \"08e67d03-f589-4064-862b-1fb49eff9bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:05.566709 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.566679    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:05.600708 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.600678    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnngq\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-kube-api-access-nnngq\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.600845 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.600715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-installation-pull-secrets\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.600845 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.600743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-bound-sa-token\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.600845 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.600765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84e3975d-dbd7-4c94-a16c-e61da230a8c7-tmp\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.600845 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.600791    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/84e3975d-dbd7-4c94-a16c-e61da230a8c7-snapshots\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.600845 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.600813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e3975d-dbd7-4c94-a16c-e61da230a8c7-trusted-ca-bundle\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.601099 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.600864    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84e3975d-dbd7-4c94-a16c-e61da230a8c7-serving-cert\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.601099 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.600893    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slf4b\" (UniqueName: \"kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b\") pod \"network-check-target-vmv47\" (UID: \"f247b77e-176f-4657-9447-ef0b89f708b3\") " pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:46:05.601099 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.600934    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-ca-trust-extracted\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.601099 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.601004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e3975d-dbd7-4c94-a16c-e61da230a8c7-service-ca-bundle\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.601099 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.601077    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.601321 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.601117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-certificates\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.601321 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.601280    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84e3975d-dbd7-4c94-a16c-e61da230a8c7-tmp\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.601501 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.601478    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-ca-trust-extracted\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.601554 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:05.601530    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:05.601554 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.601533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46hpx\" (UniqueName: \"kubernetes.io/projected/84e3975d-dbd7-4c94-a16c-e61da230a8c7-kube-api-access-46hpx\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.601554 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:05.601546    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5884f9db8b-s8cjv: secret "image-registry-tls" not found
Sep 16 18:46:05.601697 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:05.601601    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls podName:34ca68d7-921c-4d9f-bb0f-89c482e9fb17 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:06.101581965 +0000 UTC m=+65.904300175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls") pod "image-registry-5884f9db8b-s8cjv" (UID: "34ca68d7-921c-4d9f-bb0f-89c482e9fb17") : secret "image-registry-tls" not found
Sep 16 18:46:05.601697 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.601612    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/84e3975d-dbd7-4c94-a16c-e61da230a8c7-snapshots\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.601697 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.601643    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-image-registry-private-configuration\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.601697 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.601673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-trusted-ca\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.601885 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.601700    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-certificates\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.602285 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.602241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e3975d-dbd7-4c94-a16c-e61da230a8c7-service-ca-bundle\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.602673 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.602649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e3975d-dbd7-4c94-a16c-e61da230a8c7-trusted-ca-bundle\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.602764 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.602752    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-trusted-ca\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.603594 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.603571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-installation-pull-secrets\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.603678 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.603655    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84e3975d-dbd7-4c94-a16c-e61da230a8c7-serving-cert\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.604172 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.604156    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:46:05.604307 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.604291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-image-registry-private-configuration\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.610787 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.610762    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46hpx\" (UniqueName: \"kubernetes.io/projected/84e3975d-dbd7-4c94-a16c-e61da230a8c7-kube-api-access-46hpx\") pod \"insights-operator-5844b856db-f2clx\" (UID: \"84e3975d-dbd7-4c94-a16c-e61da230a8c7\") " pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.611038 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.611023    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-bound-sa-token\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.611117 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.611083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnngq\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-kube-api-access-nnngq\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:05.614323 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.614308    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:46:05.616321 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.616307    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-f2clx"
Sep 16 18:46:05.624735 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.624717    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slf4b\" (UniqueName: \"kubernetes.io/projected/f247b77e-176f-4657-9447-ef0b89f708b3-kube-api-access-slf4b\") pod \"network-check-target-vmv47\" (UID: \"f247b77e-176f-4657-9447-ef0b89f708b3\") " pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:46:05.727464 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.727434    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l6stq"]
Sep 16 18:46:05.730526 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:05.730498    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7a47e0_a2e8_4c1d_815b_18c9cecea99d.slice/crio-40b4cd9d02739a02fedf1bcafb0a78abe13f47661c32c2bccd484c8ad41a33f2 WatchSource:0}: Error finding container 40b4cd9d02739a02fedf1bcafb0a78abe13f47661c32c2bccd484c8ad41a33f2: Status 404 returned error can't find the container with id 40b4cd9d02739a02fedf1bcafb0a78abe13f47661c32c2bccd484c8ad41a33f2
Sep 16 18:46:05.750013 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.749990    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hrh8k\""
Sep 16 18:46:05.758464 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.758442    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:46:05.804106 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.804078    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz"]
Sep 16 18:46:05.806720 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:05.806691    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c177863_7189_4c95_8725_4c72172ffcf1.slice/crio-e26922127d88f65c3f903f9f521735eeeda6347f0702702274bcc00d6abd1f51 WatchSource:0}: Error finding container e26922127d88f65c3f903f9f521735eeeda6347f0702702274bcc00d6abd1f51: Status 404 returned error can't find the container with id e26922127d88f65c3f903f9f521735eeeda6347f0702702274bcc00d6abd1f51
Sep 16 18:46:05.915336 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.915305    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-f2clx"]
Sep 16 18:46:05.917946 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:05.917922    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e3975d_dbd7_4c94_a16c_e61da230a8c7.slice/crio-7abc3c6cf11c72c3e7fb8f5ad1683d967913166a1032f7018d0fe793a831b26f WatchSource:0}: Error finding container 7abc3c6cf11c72c3e7fb8f5ad1683d967913166a1032f7018d0fe793a831b26f: Status 404 returned error can't find the container with id 7abc3c6cf11c72c3e7fb8f5ad1683d967913166a1032f7018d0fe793a831b26f
Sep 16 18:46:05.998968 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.998883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz" event={"ID":"9c177863-7189-4c95-8725-4c72172ffcf1","Type":"ContainerStarted","Data":"e26922127d88f65c3f903f9f521735eeeda6347f0702702274bcc00d6abd1f51"}
Sep 16 18:46:06.000016 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:05.999991    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-f2clx" event={"ID":"84e3975d-dbd7-4c94-a16c-e61da230a8c7","Type":"ContainerStarted","Data":"7abc3c6cf11c72c3e7fb8f5ad1683d967913166a1032f7018d0fe793a831b26f"}
Sep 16 18:46:06.001149 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:06.001126    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l6stq" event={"ID":"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d","Type":"ContainerStarted","Data":"40b4cd9d02739a02fedf1bcafb0a78abe13f47661c32c2bccd484c8ad41a33f2"}
Sep 16 18:46:06.005500 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:06.005480    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-f7764\" (UID: \"08e67d03-f589-4064-862b-1fb49eff9bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:06.005561 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:06.005511    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tm8zv\" (UID: \"90969688-8134-4f01-8002-b274aed1c2da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"
Sep 16 18:46:06.005621 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:06.005609    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:46:06.005660 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:06.005615    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:06.005693 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:06.005662    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls podName:90969688-8134-4f01-8002-b274aed1c2da nodeName:}" failed. No retries permitted until 2026-09-16 18:46:07.005648219 +0000 UTC m=+66.808366414 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tm8zv" (UID: "90969688-8134-4f01-8002-b274aed1c2da") : secret "samples-operator-tls" not found
Sep 16 18:46:06.005693 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:06.005676    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls podName:08e67d03-f589-4064-862b-1fb49eff9bb2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:07.005670013 +0000 UTC m=+66.808388207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-f7764" (UID: "08e67d03-f589-4064-862b-1fb49eff9bb2") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:06.054532 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:06.054499    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vmv47"]
Sep 16 18:46:06.059024 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:06.057922    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf247b77e_176f_4657_9447_ef0b89f708b3.slice/crio-61f22d9647bd2625393ee90f3386fd226096a2bb2d5e498a97dcb92169dd4f2c WatchSource:0}: Error finding container 61f22d9647bd2625393ee90f3386fd226096a2bb2d5e498a97dcb92169dd4f2c: Status 404 returned error can't find the container with id 61f22d9647bd2625393ee90f3386fd226096a2bb2d5e498a97dcb92169dd4f2c
Sep 16 18:46:06.106008 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:06.105973    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:06.106196 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:06.106139    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:06.106196 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:06.106155    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5884f9db8b-s8cjv: secret "image-registry-tls" not found
Sep 16 18:46:06.106276 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:06.106208    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls podName:34ca68d7-921c-4d9f-bb0f-89c482e9fb17 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:07.10619326 +0000 UTC m=+66.908911459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls") pod "image-registry-5884f9db8b-s8cjv" (UID: "34ca68d7-921c-4d9f-bb0f-89c482e9fb17") : secret "image-registry-tls" not found
Sep 16 18:46:06.913199 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:06.913071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:46:06.913199 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:06.913200    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:06.913727 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:06.913269    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls podName:f6ce5cd3-f535-4b56-8a33-e47719901e55 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:38.913250936 +0000 UTC m=+98.715969147 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls") pod "dns-default-dk7qf" (UID: "f6ce5cd3-f535-4b56-8a33-e47719901e55") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:07.006969 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:07.006901    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vmv47" event={"ID":"f247b77e-176f-4657-9447-ef0b89f708b3","Type":"ContainerStarted","Data":"61f22d9647bd2625393ee90f3386fd226096a2bb2d5e498a97dcb92169dd4f2c"}
Sep 16 18:46:07.014367 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:07.013683    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-f7764\" (UID: \"08e67d03-f589-4064-862b-1fb49eff9bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:07.014367 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:07.013736    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tm8zv\" (UID: \"90969688-8134-4f01-8002-b274aed1c2da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"
Sep 16 18:46:07.014367 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:07.013872    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:46:07.014367 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:07.013928    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls podName:90969688-8134-4f01-8002-b274aed1c2da nodeName:}" failed. No retries permitted until 2026-09-16 18:46:09.013911455 +0000 UTC m=+68.816629652 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tm8zv" (UID: "90969688-8134-4f01-8002-b274aed1c2da") : secret "samples-operator-tls" not found
Sep 16 18:46:07.014367 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:07.013993    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:07.014367 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:07.014029    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls podName:08e67d03-f589-4064-862b-1fb49eff9bb2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:09.014017995 +0000 UTC m=+68.816736191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-f7764" (UID: "08e67d03-f589-4064-862b-1fb49eff9bb2") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:07.114999 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:07.114414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:07.114999 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:07.114564    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:07.114999 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:07.114581    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5884f9db8b-s8cjv: secret "image-registry-tls" not found
Sep 16 18:46:07.114999 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:07.114644    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls podName:34ca68d7-921c-4d9f-bb0f-89c482e9fb17 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:09.114625571 +0000 UTC m=+68.917343770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls") pod "image-registry-5884f9db8b-s8cjv" (UID: "34ca68d7-921c-4d9f-bb0f-89c482e9fb17") : secret "image-registry-tls" not found
Sep 16 18:46:07.568909 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:07.568866    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:07.642427 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:07.642360    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:09.016737 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:09.016429    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz" event={"ID":"9c177863-7189-4c95-8725-4c72172ffcf1","Type":"ContainerStarted","Data":"99a8ed150776da95723924c902e996f7079ac53c797f2e86d35664c9fb9238d4"}
Sep 16 18:46:09.031189 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:09.031155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tm8zv\" (UID: \"90969688-8134-4f01-8002-b274aed1c2da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"
Sep 16 18:46:09.031373 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:09.031284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-f7764\" (UID: \"08e67d03-f589-4064-862b-1fb49eff9bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:09.031373 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:09.031340    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:46:09.031373 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:09.031372    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:09.031526 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:09.031404    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls podName:90969688-8134-4f01-8002-b274aed1c2da nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.031387002 +0000 UTC m=+72.834105210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tm8zv" (UID: "90969688-8134-4f01-8002-b274aed1c2da") : secret "samples-operator-tls" not found
Sep 16 18:46:09.031526 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:09.031423    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls podName:08e67d03-f589-4064-862b-1fb49eff9bb2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.031413682 +0000 UTC m=+72.834131878 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-f7764" (UID: "08e67d03-f589-4064-862b-1fb49eff9bb2") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:09.040306 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:09.040259    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz" podStartSLOduration=1.012750362 podStartE2EDuration="4.040241104s" totalImagesPullingTime="3.027490742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:09.038815636 +0000 UTC m=+68.841533854" watchObservedRunningTime="2026-09-16 18:46:09.040241104 +0000 UTC m=+68.842959324"
Sep 16 18:46:09.132241 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:09.132006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:09.132241 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:09.132189    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:09.132241 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:09.132206    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5884f9db8b-s8cjv: secret "image-registry-tls" not found
Sep 16 18:46:09.132551 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:09.132274    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls podName:34ca68d7-921c-4d9f-bb0f-89c482e9fb17 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.132255728 +0000 UTC m=+72.934973935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls") pod "image-registry-5884f9db8b-s8cjv" (UID: "34ca68d7-921c-4d9f-bb0f-89c482e9fb17") : secret "image-registry-tls" not found
Sep 16 18:46:09.716796 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:09.716760    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:10.216166 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:10.216132    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:10.457709 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:10.457675    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:10.560596 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:10.560484    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:10.782542 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:10.782509    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:10.825544 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:10.825464    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mbgzf"]
Sep 16 18:46:10.828378 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:10.828354    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mbgzf"
Sep 16 18:46:10.831455 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:10.831265    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qdnvs\""
Sep 16 18:46:10.831455 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:10.831452    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:10.832488 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:10.832468    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:46:10.842909 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:10.842881    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mbgzf"]
Sep 16 18:46:10.887577 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:10.887542    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:10.945922 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:10.945881    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsg65\" (UniqueName: \"kubernetes.io/projected/72334f8b-d84b-4afe-a76f-80bdafdd21e5-kube-api-access-bsg65\") pod \"migrator-8b4d697c5-mbgzf\" (UID: \"72334f8b-d84b-4afe-a76f-80bdafdd21e5\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mbgzf"
Sep 16 18:46:10.988659 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:10.988627    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:11.023823 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:11.023787    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-f2clx" event={"ID":"84e3975d-dbd7-4c94-a16c-e61da230a8c7","Type":"ContainerStarted","Data":"3d5a3f61b4d0925a4302e43d3e4c144e0c9983133244beb4d660f6c94b45d717"}
Sep 16 18:46:11.043640 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:11.043583    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-f2clx" podStartSLOduration=3.126657912 podStartE2EDuration="6.043566171s" totalImagesPullingTime="2.916908259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:11.042621224 +0000 UTC m=+70.845339437" watchObservedRunningTime="2026-09-16 18:46:11.043566171 +0000 UTC m=+70.846284387"
Sep 16 18:46:11.047041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:11.047007    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsg65\" (UniqueName: \"kubernetes.io/projected/72334f8b-d84b-4afe-a76f-80bdafdd21e5-kube-api-access-bsg65\") pod \"migrator-8b4d697c5-mbgzf\" (UID: \"72334f8b-d84b-4afe-a76f-80bdafdd21e5\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mbgzf"
Sep 16 18:46:11.055639 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:11.055619    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsg65\" (UniqueName: \"kubernetes.io/projected/72334f8b-d84b-4afe-a76f-80bdafdd21e5-kube-api-access-bsg65\") pod \"migrator-8b4d697c5-mbgzf\" (UID: \"72334f8b-d84b-4afe-a76f-80bdafdd21e5\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mbgzf"
Sep 16 18:46:11.139257 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:11.139171    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mbgzf"
Sep 16 18:46:11.688854 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:11.688767    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mbgzf"]
Sep 16 18:46:11.693154 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:11.693124    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72334f8b_d84b_4afe_a76f_80bdafdd21e5.slice/crio-d304c9429e807d6cfb62fdd93421e7a63d764be9c821944079a8f3bc2b67e763 WatchSource:0}: Error finding container d304c9429e807d6cfb62fdd93421e7a63d764be9c821944079a8f3bc2b67e763: Status 404 returned error can't find the container with id d304c9429e807d6cfb62fdd93421e7a63d764be9c821944079a8f3bc2b67e763
Sep 16 18:46:12.029875 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.029835    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vmv47" event={"ID":"f247b77e-176f-4657-9447-ef0b89f708b3","Type":"ContainerStarted","Data":"19b570c15af9c1b6b99d4ff3cb21507236a54f22dbee901e7dd4b9c57b1b0785"}
Sep 16 18:46:12.030119 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.029934    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:46:12.031032 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.030993    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mbgzf" event={"ID":"72334f8b-d84b-4afe-a76f-80bdafdd21e5","Type":"ContainerStarted","Data":"d304c9429e807d6cfb62fdd93421e7a63d764be9c821944079a8f3bc2b67e763"}
Sep 16 18:46:12.032519 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.032491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l6stq" event={"ID":"0c7a47e0-a2e8-4c1d-815b-18c9cecea99d","Type":"ContainerStarted","Data":"3950e39ffc31594ac0a87083e59be0fc7dac9c5fc5aa7f6539ed8f67054f3324"}
Sep 16 18:46:12.074138 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.074080    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vmv47" podStartSLOduration=65.717339339 podStartE2EDuration="1m11.074048842s" totalImagesPullingTime="5.356709503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:12.055674591 +0000 UTC m=+71.858392808" watchObservedRunningTime="2026-09-16 18:46:12.074048842 +0000 UTC m=+71.876767058"
Sep 16 18:46:12.074652 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.074621    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l6stq" podStartSLOduration=65.38801703 podStartE2EDuration="1m11.074609545s" totalImagesPullingTime="5.686592515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:12.073367817 +0000 UTC m=+71.876086044" watchObservedRunningTime="2026-09-16 18:46:12.074609545 +0000 UTC m=+71.877327805"
Sep 16 18:46:12.078925 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.078876    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:12.370957 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.370883    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:12.381367 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.381336    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hpnfx"]
Sep 16 18:46:12.384102 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.384086    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:12.386711 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.386686    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hz5t4\""
Sep 16 18:46:12.386877 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.386846    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:46:12.386981 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.386896    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:46:12.395072 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.395035    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hpnfx"]
Sep 16 18:46:12.449588 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.449550    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:12.459491 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.459461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:12.459684 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.459494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9b9636af-e25e-4622-9345-ec8b6609b6ae-crio-socket\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:12.459684 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.459523    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:12.459684 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.459608    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbdnc\" (UniqueName: \"kubernetes.io/projected/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-api-access-hbdnc\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:12.459932 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.459715    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b9636af-e25e-4622-9345-ec8b6609b6ae-data-volume\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:12.560107 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.560042    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:12.560107 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.560110    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9b9636af-e25e-4622-9345-ec8b6609b6ae-crio-socket\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:12.560361 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.560189    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9b9636af-e25e-4622-9345-ec8b6609b6ae-crio-socket\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:12.560361 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:12.560278    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:12.560361 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.560319    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:12.560361 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:12.560360    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm podName:9b9636af-e25e-4622-9345-ec8b6609b6ae nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.060337736 +0000 UTC m=+72.863055950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hpnfx" (UID: "9b9636af-e25e-4622-9345-ec8b6609b6ae") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:12.560589 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.560409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbdnc\" (UniqueName: \"kubernetes.io/projected/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-api-access-hbdnc\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:12.560589 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:12.560428    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:12.560589 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:12.560473    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls podName:9b9636af-e25e-4622-9345-ec8b6609b6ae nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.06045628 +0000 UTC m=+72.863174479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hpnfx" (UID: "9b9636af-e25e-4622-9345-ec8b6609b6ae") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:12.560589 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.560515    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b9636af-e25e-4622-9345-ec8b6609b6ae-data-volume\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:12.560834 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.560819    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b9636af-e25e-4622-9345-ec8b6609b6ae-data-volume\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:12.570588 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:12.570566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbdnc\" (UniqueName: \"kubernetes.io/projected/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-api-access-hbdnc\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:13.063492 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:13.063452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-f7764\" (UID: \"08e67d03-f589-4064-862b-1fb49eff9bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:13.063492 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:13.063495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tm8zv\" (UID: \"90969688-8134-4f01-8002-b274aed1c2da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"
Sep 16 18:46:13.064016 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:13.063601    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:46:13.064016 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:13.063604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:13.064016 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:13.063646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:13.064016 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:13.063610    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:13.064016 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:13.063653    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:13.064016 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:13.063720    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:13.064016 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:13.063665    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls podName:90969688-8134-4f01-8002-b274aed1c2da nodeName:}" failed. No retries permitted until 2026-09-16 18:46:21.063645765 +0000 UTC m=+80.866363977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tm8zv" (UID: "90969688-8134-4f01-8002-b274aed1c2da") : secret "samples-operator-tls" not found
Sep 16 18:46:13.064016 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:13.063785    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls podName:08e67d03-f589-4064-862b-1fb49eff9bb2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:21.063756223 +0000 UTC m=+80.866474420 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-f7764" (UID: "08e67d03-f589-4064-862b-1fb49eff9bb2") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:13.064016 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:13.063840    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm podName:9b9636af-e25e-4622-9345-ec8b6609b6ae nodeName:}" failed. No retries permitted until 2026-09-16 18:46:14.063831229 +0000 UTC m=+73.866549428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hpnfx" (UID: "9b9636af-e25e-4622-9345-ec8b6609b6ae") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:13.064016 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:13.063852    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls podName:9b9636af-e25e-4622-9345-ec8b6609b6ae nodeName:}" failed. No retries permitted until 2026-09-16 18:46:14.063846475 +0000 UTC m=+73.866564668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hpnfx" (UID: "9b9636af-e25e-4622-9345-ec8b6609b6ae") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:13.164765 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:13.164740    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:13.164903 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:13.164855    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:13.164903 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:13.164868    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5884f9db8b-s8cjv: secret "image-registry-tls" not found
Sep 16 18:46:13.164971 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:13.164922    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls podName:34ca68d7-921c-4d9f-bb0f-89c482e9fb17 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:21.164909483 +0000 UTC m=+80.967627677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls") pod "image-registry-5884f9db8b-s8cjv" (UID: "34ca68d7-921c-4d9f-bb0f-89c482e9fb17") : secret "image-registry-tls" not found
Sep 16 18:46:13.216610 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:13.216583    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:13.895721 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:13.895683    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:14.025516 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:14.025468    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:14.041611 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:14.041572    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mbgzf" event={"ID":"72334f8b-d84b-4afe-a76f-80bdafdd21e5","Type":"ContainerStarted","Data":"0dff7b682c32d34bd2bcc9976c59d16f61d40b5fb53ba56c9f73dedce890617e"}
Sep 16 18:46:14.041752 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:14.041620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mbgzf" event={"ID":"72334f8b-d84b-4afe-a76f-80bdafdd21e5","Type":"ContainerStarted","Data":"e8d0786ca13e95f0e3844f803b02ca2af1cc42c1988f1138edff460d47072a7a"}
Sep 16 18:46:14.059420 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:14.059367    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mbgzf" podStartSLOduration=2.639189934 podStartE2EDuration="4.059351835s" totalImagesPullingTime="1.420161901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:14.058275947 +0000 UTC m=+73.860994164" watchObservedRunningTime="2026-09-16 18:46:14.059351835 +0000 UTC m=+73.862070089"
Sep 16 18:46:14.073165 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:14.073138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:14.073512 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:14.073175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:14.073512 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:14.073271    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:14.073512 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:14.073311    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:14.073512 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:14.073327    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm podName:9b9636af-e25e-4622-9345-ec8b6609b6ae nodeName:}" failed. No retries permitted until 2026-09-16 18:46:16.073312342 +0000 UTC m=+75.876030536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hpnfx" (UID: "9b9636af-e25e-4622-9345-ec8b6609b6ae") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:14.073512 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:14.073362    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls podName:9b9636af-e25e-4622-9345-ec8b6609b6ae nodeName:}" failed. No retries permitted until 2026-09-16 18:46:16.073347929 +0000 UTC m=+75.876066141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hpnfx" (UID: "9b9636af-e25e-4622-9345-ec8b6609b6ae") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:14.168325 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:14.168239    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:14.270103 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:14.270070    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:14.415507 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:14.415476    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:14.518833 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:14.518807    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:14.636475 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:14.636447    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:15.127958 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:15.127932    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hm84r_0ca39171-efab-4995-a0fb-2261a074d3c9/dns-node-resolver/0.log"
Sep 16 18:46:15.362772 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:15.362740    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:15.767318 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:15.767289    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:15.928284 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:15.928255    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ldqrg_a5223494-bd2b-4911-ae88-8bb10b49b6f5/node-ca/0.log"
Sep 16 18:46:16.090536 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:16.090460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:16.090536 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:16.090503    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:16.090731 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:16.090581    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:16.090731 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:16.090646    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm podName:9b9636af-e25e-4622-9345-ec8b6609b6ae nodeName:}" failed. No retries permitted until 2026-09-16 18:46:20.090630824 +0000 UTC m=+79.893349017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hpnfx" (UID: "9b9636af-e25e-4622-9345-ec8b6609b6ae") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:16.090731 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:16.090652    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:16.090731 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:16.090691    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls podName:9b9636af-e25e-4622-9345-ec8b6609b6ae nodeName:}" failed. No retries permitted until 2026-09-16 18:46:20.09067933 +0000 UTC m=+79.893397524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hpnfx" (UID: "9b9636af-e25e-4622-9345-ec8b6609b6ae") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:16.210207 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:16.210170    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:16.567812 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:16.567776    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:16.691365 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:16.691331    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:16.939907 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:16.939830    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kthn2_ae49ada0-3615-4001-8c5d-4fa78bbda02d/serve-healthcheck-canary/0.log"
Sep 16 18:46:16.986495 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:16.986466    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:17.169031 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:17.168999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:17.329357 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:17.329327    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-mbgzf_72334f8b-d84b-4afe-a76f-80bdafdd21e5/migrator/0.log"
Sep 16 18:46:17.529101 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:17.529051    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-mbgzf_72334f8b-d84b-4afe-a76f-80bdafdd21e5/graceful-termination/0.log"
Sep 16 18:46:17.728858 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:17.728820    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-p8frz_9c177863-7189-4c95-8725-4c72172ffcf1/kube-storage-version-migrator-operator/0.log"
Sep 16 18:46:17.755458 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:17.755428    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:17.855213 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:17.855179    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:18.216554 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:18.216513    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:18.636274 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:18.636193    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:18.770704 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:18.770675    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:18.854989 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:18.854957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:19.187756 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:19.187720    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:19.775890 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:19.775861    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:20.122207 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:20.122116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:20.122207 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:20.122159    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:20.122411 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:20.122265    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:20.122411 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:20.122292    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:20.122411 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:20.122330    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm podName:9b9636af-e25e-4622-9345-ec8b6609b6ae nodeName:}" failed. No retries permitted until 2026-09-16 18:46:28.122315667 +0000 UTC m=+87.925033861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hpnfx" (UID: "9b9636af-e25e-4622-9345-ec8b6609b6ae") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:20.122411 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:20.122345    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls podName:9b9636af-e25e-4622-9345-ec8b6609b6ae nodeName:}" failed. No retries permitted until 2026-09-16 18:46:28.122339211 +0000 UTC m=+87.925057405 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hpnfx" (UID: "9b9636af-e25e-4622-9345-ec8b6609b6ae") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:20.224068 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:20.224029    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:20.392981 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:20.392897    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:20.618948 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:20.618914    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:21.040443 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.040412    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:21.130985 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.130838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-f7764\" (UID: \"08e67d03-f589-4064-862b-1fb49eff9bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:21.130985 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.130887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tm8zv\" (UID: \"90969688-8134-4f01-8002-b274aed1c2da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"
Sep 16 18:46:21.133486 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.133454    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08e67d03-f589-4064-862b-1fb49eff9bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-f7764\" (UID: \"08e67d03-f589-4064-862b-1fb49eff9bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:21.133610 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.133513    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90969688-8134-4f01-8002-b274aed1c2da-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tm8zv\" (UID: \"90969688-8134-4f01-8002-b274aed1c2da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"
Sep 16 18:46:21.191797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.191760    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"
Sep 16 18:46:21.231908 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.231864    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:21.234632 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.234602    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls\") pod \"image-registry-5884f9db8b-s8cjv\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") " pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:21.287098 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.287033    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:21.416388 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.416337    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"
Sep 16 18:46:21.419164 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.419134    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:21.472642 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.472555    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764"]
Sep 16 18:46:21.474578 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:21.474552    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e67d03_f589_4064_862b_1fb49eff9bb2.slice/crio-6959e520a0f11324ad029ead40800cee47c70ca85bed9683830c473e81e32028 WatchSource:0}: Error finding container 6959e520a0f11324ad029ead40800cee47c70ca85bed9683830c473e81e32028: Status 404 returned error can't find the container with id 6959e520a0f11324ad029ead40800cee47c70ca85bed9683830c473e81e32028
Sep 16 18:46:21.588779 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.588702    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5884f9db8b-s8cjv"]
Sep 16 18:46:21.591871 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:21.591846    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ca68d7_921c_4d9f_bb0f_89c482e9fb17.slice/crio-9085d5a364ea4827f28d68ae4ee38503b21e94b32c49533a9e91b4e8a31800f9 WatchSource:0}: Error finding container 9085d5a364ea4827f28d68ae4ee38503b21e94b32c49533a9e91b4e8a31800f9: Status 404 returned error can't find the container with id 9085d5a364ea4827f28d68ae4ee38503b21e94b32c49533a9e91b4e8a31800f9
Sep 16 18:46:21.701105 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.701045    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv"]
Sep 16 18:46:21.706451 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.706409    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:21.810331 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:21.810296    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:22.074169 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:22.073975    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv" event={"ID":"34ca68d7-921c-4d9f-bb0f-89c482e9fb17","Type":"ContainerStarted","Data":"37bc34d793aacad72563b45b719d5d46e97b31be9c6db038c480dce123a55726"}
Sep 16 18:46:22.074169 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:22.074038    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv" event={"ID":"34ca68d7-921c-4d9f-bb0f-89c482e9fb17","Type":"ContainerStarted","Data":"9085d5a364ea4827f28d68ae4ee38503b21e94b32c49533a9e91b4e8a31800f9"}
Sep 16 18:46:22.074169 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:22.074081    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:22.075402 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:22.075376    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764" event={"ID":"08e67d03-f589-4064-862b-1fb49eff9bb2","Type":"ContainerStarted","Data":"6959e520a0f11324ad029ead40800cee47c70ca85bed9683830c473e81e32028"}
Sep 16 18:46:22.076545 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:22.076520    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv" event={"ID":"90969688-8134-4f01-8002-b274aed1c2da","Type":"ContainerStarted","Data":"ded567f49c5e8f97a4c450f877910e675bfe741c6cbc33f4187e9e115c903c0e"}
Sep 16 18:46:22.794198 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:22.794127    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:22.997883 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:22.997844    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.361760 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:23.361723    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.456664 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:23.456618    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.601286 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:23.601235    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.875813 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:23.875775    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.972788 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:23.972750    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:24.084394 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:24.084356    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:24.085573 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:24.085544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764" event={"ID":"08e67d03-f589-4064-862b-1fb49eff9bb2","Type":"ContainerStarted","Data":"413dcf8e8ef9cdb6a7403c41909981d91c2d9a348d5cda22977650aad6433d1f"}
Sep 16 18:46:24.087474 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:24.087450    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv" event={"ID":"90969688-8134-4f01-8002-b274aed1c2da","Type":"ContainerStarted","Data":"9b1dfe6d720f2af67c3e212dc430722b42da204a5eac18bed592761e86ec8741"}
Sep 16 18:46:24.087578 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:24.087498    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv" event={"ID":"90969688-8134-4f01-8002-b274aed1c2da","Type":"ContainerStarted","Data":"16771c5e70e83bdc5eb93bba7fc6a4b53ee52f87998585e18c19ceb1fc1ea676"}
Sep 16 18:46:24.119092 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:24.119024    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-f7764" podStartSLOduration=16.893602211 podStartE2EDuration="19.119006175s" totalImagesPullingTime="2.225403964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:24.118320989 +0000 UTC m=+83.921039206" watchObservedRunningTime="2026-09-16 18:46:24.119006175 +0000 UTC m=+83.921724381"
Sep 16 18:46:24.119265 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:24.119135    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv" podStartSLOduration=19.119127502 podStartE2EDuration="19.119127502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:22.111021204 +0000 UTC m=+81.913739419" watchObservedRunningTime="2026-09-16 18:46:24.119127502 +0000 UTC m=+83.921845734"
Sep 16 18:46:24.143827 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:24.143773    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tm8zv" podStartSLOduration=17.180654802 podStartE2EDuration="19.143734054s" totalImagesPullingTime="1.963079252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:24.142923204 +0000 UTC m=+83.945641419" watchObservedRunningTime="2026-09-16 18:46:24.143734054 +0000 UTC m=+83.946452319"
Sep 16 18:46:24.300868 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:24.300784    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:24.408679 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:24.408650    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:24.682978 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:24.682903    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:25.136376 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:25.136346    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:25.258334 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:25.258302    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:28.189154 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:28.189102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:28.189154 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:28.189160    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:28.189660 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:28.189640    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b9636af-e25e-4622-9345-ec8b6609b6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:28.191622 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:28.191605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b9636af-e25e-4622-9345-ec8b6609b6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hpnfx\" (UID: \"9b9636af-e25e-4622-9345-ec8b6609b6ae\") " pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:28.293495 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:28.293459    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hpnfx"
Sep 16 18:46:28.565781 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:28.565646    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hpnfx"]
Sep 16 18:46:28.568435 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:28.568392    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9636af_e25e_4622_9345_ec8b6609b6ae.slice/crio-9d8afa7d3a1a28c7ffbd85756cee58dc447b2981c741f1eedde1295957163bc8 WatchSource:0}: Error finding container 9d8afa7d3a1a28c7ffbd85756cee58dc447b2981c741f1eedde1295957163bc8: Status 404 returned error can't find the container with id 9d8afa7d3a1a28c7ffbd85756cee58dc447b2981c741f1eedde1295957163bc8
Sep 16 18:46:29.105689 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:29.105650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hpnfx" event={"ID":"9b9636af-e25e-4622-9345-ec8b6609b6ae","Type":"ContainerStarted","Data":"f7004e9a7d8a9ba64d2a61228aa5ec64336c39ec2746f6e1f9bc9dd5f01595b9"}
Sep 16 18:46:29.105865 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:29.105729    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hpnfx" event={"ID":"9b9636af-e25e-4622-9345-ec8b6609b6ae","Type":"ContainerStarted","Data":"9d8afa7d3a1a28c7ffbd85756cee58dc447b2981c741f1eedde1295957163bc8"}
Sep 16 18:46:29.771559 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:29.771517    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:29.877598 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:29.877566    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:30.082000 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:30.081912    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:30.111637 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:30.111586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hpnfx" event={"ID":"9b9636af-e25e-4622-9345-ec8b6609b6ae","Type":"ContainerStarted","Data":"2716878cf2f82541f31faf41ebab97454727b66ffbab8a318d3e6c2e74b69bca"}
Sep 16 18:46:30.260647 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:30.260609    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:30.340008 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:30.339928    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:30.454515 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:30.454479    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:30.862021 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:30.861988    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:30.936364 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:30.936336    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.084328 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:31.084294    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.116624 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:31.116544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hpnfx" event={"ID":"9b9636af-e25e-4622-9345-ec8b6609b6ae","Type":"ContainerStarted","Data":"fa2220162ed1a3c4e58d749cd47f72c57012867b93c45c6b0dcb59f97751e6da"}
Sep 16 18:46:31.144331 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:31.144279    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hpnfx" podStartSLOduration=17.471027283 podStartE2EDuration="19.144264373s" totalImagesPullingTime="1.67323709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:31.142466027 +0000 UTC m=+90.945184243" watchObservedRunningTime="2026-09-16 18:46:31.144264373 +0000 UTC m=+90.946982588"
Sep 16 18:46:31.284486 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:31.284450    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.465401 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:31.465366    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.648217 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:31.648186    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.988222 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:31.988190    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:33.473537 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:33.473491    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:33.573565 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:33.573536    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:34.620594 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.620563    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5884f9db8b-s8cjv"]
Sep 16 18:46:34.666669 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.666637    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f5dcf5979-vwfhs"]
Sep 16 18:46:34.687088 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.682620    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.692889 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.692856    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f5dcf5979-vwfhs"]
Sep 16 18:46:34.739970 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.739938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-installation-pull-secrets\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.740166 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.739977    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-image-registry-private-configuration\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.740166 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.740042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-trusted-ca\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.740166 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.740088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-registry-certificates\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.740166 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.740109    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-ca-trust-extracted\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.740166 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.740125    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-bound-sa-token\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.740166 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.740142    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llpkd\" (UniqueName: \"kubernetes.io/projected/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-kube-api-access-llpkd\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.740398 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.740180    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-registry-tls\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.748798 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.748774    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:34.841123 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.841080    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-trusted-ca\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.841123 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.841127    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-registry-certificates\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.841387 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.841146    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-ca-trust-extracted\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.841387 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.841163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-bound-sa-token\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.841387 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.841178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llpkd\" (UniqueName: \"kubernetes.io/projected/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-kube-api-access-llpkd\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.841387 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.841195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-registry-tls\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.841387 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.841241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-installation-pull-secrets\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.841387 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.841277    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-image-registry-private-configuration\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.841750 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.841724    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-ca-trust-extracted\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.842091 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.842031    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-registry-certificates\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.842299 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.842281    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-trusted-ca\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.843893 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.843872    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-installation-pull-secrets\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.843982 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.843961    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-registry-tls\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.844036 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.843967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-image-registry-private-configuration\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.852339 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.852317    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llpkd\" (UniqueName: \"kubernetes.io/projected/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-kube-api-access-llpkd\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.852465 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.852448    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f-bound-sa-token\") pod \"image-registry-6f5dcf5979-vwfhs\" (UID: \"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f\") " pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:34.860155 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.860127    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:34.958224 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.958139    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:34.993685 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:34.993651    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:35.279648 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:35.279602    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f5dcf5979-vwfhs"]
Sep 16 18:46:35.282331 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:35.282304    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3d99f9_a0f8_47fd_aa72_a98d5ce1c72f.slice/crio-66fb1352a2f22de1be764235a5dc6d545489abea3dde192c79b168cf436747df WatchSource:0}: Error finding container 66fb1352a2f22de1be764235a5dc6d545489abea3dde192c79b168cf436747df: Status 404 returned error can't find the container with id 66fb1352a2f22de1be764235a5dc6d545489abea3dde192c79b168cf436747df
Sep 16 18:46:36.134340 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:36.134300    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs" event={"ID":"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f","Type":"ContainerStarted","Data":"fd8b65016287c6b2654a07e6874a3eb7e9f0366d698512b47e62b82495843b2a"}
Sep 16 18:46:36.134800 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:36.134356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs" event={"ID":"ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f","Type":"ContainerStarted","Data":"66fb1352a2f22de1be764235a5dc6d545489abea3dde192c79b168cf436747df"}
Sep 16 18:46:36.134800 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:36.134421    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:36.157988 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:36.157924    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs" podStartSLOduration=2.157900279 podStartE2EDuration="2.157900279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:36.157003812 +0000 UTC m=+95.959722063" watchObservedRunningTime="2026-09-16 18:46:36.157900279 +0000 UTC m=+95.960618497"
Sep 16 18:46:38.330295 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:38.330260    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:38.464548 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:38.464514    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:38.736857 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:38.736830    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:38.972821 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:38.972776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:46:38.975257 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:38.975235    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce5cd3-f535-4b56-8a33-e47719901e55-metrics-tls\") pod \"dns-default-dk7qf\" (UID: \"f6ce5cd3-f535-4b56-8a33-e47719901e55\") " pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:46:39.242848 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:39.242818    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dwfv6\""
Sep 16 18:46:39.250448 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:39.250423    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:46:39.544364 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:39.544282    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dk7qf"]
Sep 16 18:46:39.547315 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:39.547269    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ce5cd3_f535_4b56_8a33_e47719901e55.slice/crio-755e453df435a32ed2ecfee9ea1365b91c68f83231b86d2448f3defdd6ba9bb3 WatchSource:0}: Error finding container 755e453df435a32ed2ecfee9ea1365b91c68f83231b86d2448f3defdd6ba9bb3: Status 404 returned error can't find the container with id 755e453df435a32ed2ecfee9ea1365b91c68f83231b86d2448f3defdd6ba9bb3
Sep 16 18:46:40.150001 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:40.149961    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dk7qf" event={"ID":"f6ce5cd3-f535-4b56-8a33-e47719901e55","Type":"ContainerStarted","Data":"755e453df435a32ed2ecfee9ea1365b91c68f83231b86d2448f3defdd6ba9bb3"}
Sep 16 18:46:40.277230 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:40.277195    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:40.338821 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:40.338786    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:41.471993 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:41.471957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:42.159425 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:42.159343    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dk7qf" event={"ID":"f6ce5cd3-f535-4b56-8a33-e47719901e55","Type":"ContainerStarted","Data":"c8d060cf09d45d813791bab71d7e6d28ef393b2ec69bcf077bf6eac04e541664"}
Sep 16 18:46:42.159425 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:42.159421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dk7qf" event={"ID":"f6ce5cd3-f535-4b56-8a33-e47719901e55","Type":"ContainerStarted","Data":"be3dde12269d45086c8cf919bb1c903cf21c237588839cc93afe1c40fddbb696"}
Sep 16 18:46:42.159593 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:42.159501    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:46:42.179960 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:42.179907    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dk7qf" podStartSLOduration=65.149823248 podStartE2EDuration="1m7.179890092s" totalImagesPullingTime="2.030066844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:42.179046699 +0000 UTC m=+101.981764917" watchObservedRunningTime="2026-09-16 18:46:42.179890092 +0000 UTC m=+101.982608300"
Sep 16 18:46:43.039438 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:43.039406    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vmv47"
Sep 16 18:46:43.079866 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:43.079837    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:43.174226 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:43.174196    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:43.574284 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:43.574252    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:43.907409 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:43.907328    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:43.963444 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:43.963407    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"]
Sep 16 18:46:43.994714 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:43.994684    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mbf95"]
Sep 16 18:46:43.994863 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:43.994829    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:43.997787 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:43.997764    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 18:46:43.997932 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:43.997893    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-f7q45\""
Sep 16 18:46:43.998490 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:43.998466    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:46:43.999271 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:43.999255    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:46:43.999360 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:43.999299    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 18:46:44.008739 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.008709    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"]
Sep 16 18:46:44.008870 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.008840    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.012516 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.012491    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:46:44.013049 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.013029    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gf67p\""
Sep 16 18:46:44.013177 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.013033    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:46:44.013177 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.013033    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:46:44.030744 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.030721    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:44.113694 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.113658    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:44.117844 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.117826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fa541550-1d07-47f5-ad29-44b4241abd11-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.117929 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.117852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mprcr\" (UniqueName: \"kubernetes.io/projected/fa541550-1d07-47f5-ad29-44b4241abd11-kube-api-access-mprcr\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.117929 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.117879    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d334d6df-acdd-48fc-af0f-484c6e26dc7c-root\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.117929 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.117895    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa541550-1d07-47f5-ad29-44b4241abd11-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.117929 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.117912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d334d6df-acdd-48fc-af0f-484c6e26dc7c-sys\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.118133 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.117949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-tls\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.118133 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.117970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa541550-1d07-47f5-ad29-44b4241abd11-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.118133 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.118071    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa541550-1d07-47f5-ad29-44b4241abd11-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.118133 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.118108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.118333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.118150    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d334d6df-acdd-48fc-af0f-484c6e26dc7c-metrics-client-ca\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.118333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.118179    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-textfile\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.118333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.118217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.118333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.118245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fa541550-1d07-47f5-ad29-44b4241abd11-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.118333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.118286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-wtmp\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.118333 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.118324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8wbz\" (UniqueName: \"kubernetes.io/projected/d334d6df-acdd-48fc-af0f-484c6e26dc7c-kube-api-access-k8wbz\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.216121 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.216083    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:44.219248 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fa541550-1d07-47f5-ad29-44b4241abd11-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.219336 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219255    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mprcr\" (UniqueName: \"kubernetes.io/projected/fa541550-1d07-47f5-ad29-44b4241abd11-kube-api-access-mprcr\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.219336 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219275    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d334d6df-acdd-48fc-af0f-484c6e26dc7c-root\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.219336 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa541550-1d07-47f5-ad29-44b4241abd11-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.219336 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d334d6df-acdd-48fc-af0f-484c6e26dc7c-sys\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.219536 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219375    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d334d6df-acdd-48fc-af0f-484c6e26dc7c-sys\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.219536 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219378    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d334d6df-acdd-48fc-af0f-484c6e26dc7c-root\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.219536 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-tls\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.219536 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa541550-1d07-47f5-ad29-44b4241abd11-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.219536 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219485    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa541550-1d07-47f5-ad29-44b4241abd11-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.219777 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.219777 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219648    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d334d6df-acdd-48fc-af0f-484c6e26dc7c-metrics-client-ca\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.219777 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-textfile\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.219777 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219713    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fa541550-1d07-47f5-ad29-44b4241abd11-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.219777 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219725    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.219777 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219772    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fa541550-1d07-47f5-ad29-44b4241abd11-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.220098 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-wtmp\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.220098 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.219875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8wbz\" (UniqueName: \"kubernetes.io/projected/d334d6df-acdd-48fc-af0f-484c6e26dc7c-kube-api-access-k8wbz\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.220568 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.220264    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa541550-1d07-47f5-ad29-44b4241abd11-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.220568 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.220406    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-wtmp\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.220568 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.220527    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.220783 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.220566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d334d6df-acdd-48fc-af0f-484c6e26dc7c-metrics-client-ca\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.220783 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.220631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-textfile\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.220875 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.220784    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fa541550-1d07-47f5-ad29-44b4241abd11-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.222668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.222649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa541550-1d07-47f5-ad29-44b4241abd11-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.222821 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.222802    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa541550-1d07-47f5-ad29-44b4241abd11-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.222871 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.222826    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.223208 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.223190    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d334d6df-acdd-48fc-af0f-484c6e26dc7c-node-exporter-tls\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.228220 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.228196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mprcr\" (UniqueName: \"kubernetes.io/projected/fa541550-1d07-47f5-ad29-44b4241abd11-kube-api-access-mprcr\") pod \"kube-state-metrics-59fbdd77d-4hcsj\" (UID: \"fa541550-1d07-47f5-ad29-44b4241abd11\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.228840 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.228822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8wbz\" (UniqueName: \"kubernetes.io/projected/d334d6df-acdd-48fc-af0f-484c6e26dc7c-kube-api-access-k8wbz\") pod \"node-exporter-mbf95\" (UID: \"d334d6df-acdd-48fc-af0f-484c6e26dc7c\") " pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.304016 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.303976    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"
Sep 16 18:46:44.314111 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.314033    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:44.317466 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.317444    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mbf95"
Sep 16 18:46:44.328869 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:44.328832    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd334d6df_acdd_48fc_af0f_484c6e26dc7c.slice/crio-d4d71f8898f04a96ef5d7b976fce9b1f53fc3e83c1adfab25fef3e00ae33dd28 WatchSource:0}: Error finding container d4d71f8898f04a96ef5d7b976fce9b1f53fc3e83c1adfab25fef3e00ae33dd28: Status 404 returned error can't find the container with id d4d71f8898f04a96ef5d7b976fce9b1f53fc3e83c1adfab25fef3e00ae33dd28
Sep 16 18:46:44.556145 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.556030    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:44.578977 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.578950    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj"]
Sep 16 18:46:44.582225 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:44.582188    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa541550_1d07_47f5_ad29_44b4241abd11.slice/crio-bdaae184de375695eea9c2ee1ab54b98e73dc750cd8f3f735c79138ce62410dd WatchSource:0}: Error finding container bdaae184de375695eea9c2ee1ab54b98e73dc750cd8f3f735c79138ce62410dd: Status 404 returned error can't find the container with id bdaae184de375695eea9c2ee1ab54b98e73dc750cd8f3f735c79138ce62410dd
Sep 16 18:46:44.629853 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.629805    2561 patch_prober.go:28] interesting pod/image-registry-5884f9db8b-s8cjv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:46:44.630012 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.629863    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv" podUID="34ca68d7-921c-4d9f-bb0f-89c482e9fb17" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:46:44.814678 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.814594    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:44.910814 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.910780    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:44.915750 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.915710    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:44.918946 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.918926    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:44.922171 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.922141    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:46:44.922782 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.922752    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:46:44.922901 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.922875    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:46:44.923110 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.923090    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:46:44.923219 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.923138    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6thqs\""
Sep 16 18:46:44.923282 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.923260    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:46:44.923336 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.923307    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:46:44.923391 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.923376    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:46:44.923837 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.923819    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:46:44.924102 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.924081    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:46:44.938651 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:44.938624    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:45.027242 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.027192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-config-out\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.027242 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.027238    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.027482 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.027267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59fdl\" (UniqueName: \"kubernetes.io/projected/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-kube-api-access-59fdl\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.027482 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.027370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.027482 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.027406    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.027482 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.027466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.027630 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.027488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.027630 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.027560    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.027630 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.027586    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-web-config\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.027630 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.027615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.027806 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.027652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.027806 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.027683    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.027806 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.027705    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.128589 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.128505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.128589 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.128556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59fdl\" (UniqueName: \"kubernetes.io/projected/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-kube-api-access-59fdl\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.129018 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.128595    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.129018 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.128620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.129018 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.128660    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.129018 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.128681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.129018 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.128722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.129018 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.128747    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-web-config\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.129018 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.128782    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.129018 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.128819    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.129018 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.128844    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.129018 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.128868    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.129018 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.128909    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-config-out\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.129643 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:45.129077    2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 16 18:46:45.129643 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:46:45.129150    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-main-tls podName:2b030fbd-db7e-4c90-94e0-83ceb26dbd93 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:45.62912928 +0000 UTC m=+105.431847490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "2b030fbd-db7e-4c90-94e0-83ceb26dbd93") : secret "alertmanager-main-tls" not found
Sep 16 18:46:45.129643 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.129303    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.129853 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.129816    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.129983 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.129860    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.133416 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.133076    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-config-out\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.133416 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.133359    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-web-config\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.133416 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.133384    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.133637 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.133614    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.134202 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.134181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.134425 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.134394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.134803 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.134779    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.135849 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.135786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.137674 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.137648    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59fdl\" (UniqueName: \"kubernetes.io/projected/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-kube-api-access-59fdl\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.172004 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.171967    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj" event={"ID":"fa541550-1d07-47f5-ad29-44b4241abd11","Type":"ContainerStarted","Data":"bdaae184de375695eea9c2ee1ab54b98e73dc750cd8f3f735c79138ce62410dd"}
Sep 16 18:46:45.174038 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.173999    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbf95" event={"ID":"d334d6df-acdd-48fc-af0f-484c6e26dc7c","Type":"ContainerStarted","Data":"d4d71f8898f04a96ef5d7b976fce9b1f53fc3e83c1adfab25fef3e00ae33dd28"}
Sep 16 18:46:45.247342 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.247304    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:45.348650 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.348569    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:45.634226 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.634186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.637126 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.637097    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.793830 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.793783    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:45.831710 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.831675    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:45.897098 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:45.897046    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:46.169152 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:46.169123    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b030fbd_db7e_4c90_94e0_83ceb26dbd93.slice/crio-675b290756526a72fba7f8f6c616a21225fc6ac83d65c16dc67da786a9f675c6 WatchSource:0}: Error finding container 675b290756526a72fba7f8f6c616a21225fc6ac83d65c16dc67da786a9f675c6: Status 404 returned error can't find the container with id 675b290756526a72fba7f8f6c616a21225fc6ac83d65c16dc67da786a9f675c6
Sep 16 18:46:46.169455 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:46.169317    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:46.179713 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:46.179683    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerStarted","Data":"675b290756526a72fba7f8f6c616a21225fc6ac83d65c16dc67da786a9f675c6"}
Sep 16 18:46:46.182296 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:46.182249    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj" event={"ID":"fa541550-1d07-47f5-ad29-44b4241abd11","Type":"ContainerStarted","Data":"741cc0595b1483a5a4b03a59bb68ef66094ff57b5574878a0ce40924b7702a61"}
Sep 16 18:46:46.182421 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:46.182338    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj" event={"ID":"fa541550-1d07-47f5-ad29-44b4241abd11","Type":"ContainerStarted","Data":"3c6adc13374a4498d8364a05a0be0360ea636b5ba5bfb3519e565c1eabfb8d14"}
Sep 16 18:46:46.182421 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:46.182375    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj" event={"ID":"fa541550-1d07-47f5-ad29-44b4241abd11","Type":"ContainerStarted","Data":"d8fb428767c408b5a315834698f3a9a8f211a584927ddb3747860a0e3873e3d6"}
Sep 16 18:46:46.185507 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:46.185466    2561 generic.go:356] "Generic (PLEG): container finished" podID="d334d6df-acdd-48fc-af0f-484c6e26dc7c" containerID="f65d008046d60a9e0768ea5b2a02ee5ba2470798253fbb89e0b804b70c6b53aa" exitCode=0
Sep 16 18:46:46.185635 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:46.185531    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbf95" event={"ID":"d334d6df-acdd-48fc-af0f-484c6e26dc7c","Type":"ContainerDied","Data":"f65d008046d60a9e0768ea5b2a02ee5ba2470798253fbb89e0b804b70c6b53aa"}
Sep 16 18:46:46.216190 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:46.216123    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4hcsj" podStartSLOduration=1.8931106930000001 podStartE2EDuration="3.216101944s" totalImagesPullingTime="1.322991251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:46.215266936 +0000 UTC m=+106.017985151" watchObservedRunningTime="2026-09-16 18:46:46.216101944 +0000 UTC m=+106.018820175"
Sep 16 18:46:46.811154 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:46.811121    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:46.889699 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:46.889651    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:46.953666 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:46.953622    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:47.193310 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:47.193222    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbf95" event={"ID":"d334d6df-acdd-48fc-af0f-484c6e26dc7c","Type":"ContainerStarted","Data":"e7177edc1e92b22efc353b14a74b9fa96048bfaa980f38826fe4ed7d45afa7ea"}
Sep 16 18:46:47.193310 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:47.193288    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbf95" event={"ID":"d334d6df-acdd-48fc-af0f-484c6e26dc7c","Type":"ContainerStarted","Data":"4830bfe39be245b6cc5eb987ce790a1380cd2c0aeaa8be4a5baa42fb051ee793"}
Sep 16 18:46:47.219208 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:47.219154    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mbf95" podStartSLOduration=3.077215586 podStartE2EDuration="4.219136119s" totalImagesPullingTime="1.061692898s" totalInitContainerRuntime="80.227635ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:47.217236013 +0000 UTC m=+107.019954230" watchObservedRunningTime="2026-09-16 18:46:47.219136119 +0000 UTC m=+107.021854335"
Sep 16 18:46:47.249572 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:47.249538    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:47.662049 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:47.662017    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:47.718870 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:47.718834    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:48.199189 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.199157    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerID="464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d" exitCode=0
Sep 16 18:46:48.199574 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.199249    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerDied","Data":"464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d"}
Sep 16 18:46:48.311528 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.311495    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-69566c86c6-gt6sm"]
Sep 16 18:46:48.313769 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.313752    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.318400 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.318376    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:46:48.318781 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.318761    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lgl66\""
Sep 16 18:46:48.319159 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.319141    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ehu7djic2erdn\""
Sep 16 18:46:48.319241 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.319216    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:46:48.319241 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.319228    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:46:48.321627 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.321610    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:46:48.337444 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.337408    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69566c86c6-gt6sm"]
Sep 16 18:46:48.461481 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.461387    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/81a18786-abe0-4f45-80e9-040596e64ec6-secret-metrics-server-tls\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.461481 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.461424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/81a18786-abe0-4f45-80e9-040596e64ec6-audit-log\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.461481 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.461452    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6mj6\" (UniqueName: \"kubernetes.io/projected/81a18786-abe0-4f45-80e9-040596e64ec6-kube-api-access-k6mj6\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.461713 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.461546    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/81a18786-abe0-4f45-80e9-040596e64ec6-client-ca-bundle\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.461713 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.461583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81a18786-abe0-4f45-80e9-040596e64ec6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.461713 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.461619    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/81a18786-abe0-4f45-80e9-040596e64ec6-metrics-server-audit-profiles\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.461713 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.461657    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/81a18786-abe0-4f45-80e9-040596e64ec6-secret-metrics-server-client-certs\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.548236 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.548209    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:48.563039 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.562993    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/81a18786-abe0-4f45-80e9-040596e64ec6-metrics-server-audit-profiles\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.563232 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.563045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/81a18786-abe0-4f45-80e9-040596e64ec6-secret-metrics-server-client-certs\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.563232 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.563106    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/81a18786-abe0-4f45-80e9-040596e64ec6-secret-metrics-server-tls\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.563232 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.563133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/81a18786-abe0-4f45-80e9-040596e64ec6-audit-log\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.563232 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.563175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6mj6\" (UniqueName: \"kubernetes.io/projected/81a18786-abe0-4f45-80e9-040596e64ec6-kube-api-access-k6mj6\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.563417 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.563246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/81a18786-abe0-4f45-80e9-040596e64ec6-client-ca-bundle\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.563417 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.563280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81a18786-abe0-4f45-80e9-040596e64ec6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.563613 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.563575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/81a18786-abe0-4f45-80e9-040596e64ec6-audit-log\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.564102 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.564081    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/81a18786-abe0-4f45-80e9-040596e64ec6-metrics-server-audit-profiles\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.564188 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.564151    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81a18786-abe0-4f45-80e9-040596e64ec6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.565701 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.565671    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/81a18786-abe0-4f45-80e9-040596e64ec6-secret-metrics-server-tls\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.565813 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.565730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/81a18786-abe0-4f45-80e9-040596e64ec6-secret-metrics-server-client-certs\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.565813 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.565799    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/81a18786-abe0-4f45-80e9-040596e64ec6-client-ca-bundle\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.587642 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.587611    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6mj6\" (UniqueName: \"kubernetes.io/projected/81a18786-abe0-4f45-80e9-040596e64ec6-kube-api-access-k6mj6\") pod \"metrics-server-69566c86c6-gt6sm\" (UID: \"81a18786-abe0-4f45-80e9-040596e64ec6\") " pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.622663 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.622620    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:46:48.645944 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.645915    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:48.891460 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.891419    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:48.903218 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:48.903191    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69566c86c6-gt6sm"]
Sep 16 18:46:48.903992 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:48.903964    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a18786_abe0_4f45_80e9_040596e64ec6.slice/crio-67d71cc4c0fb1981a4f97a92a60388efc4858d2a0d84b11e789435aa1eab903a WatchSource:0}: Error finding container 67d71cc4c0fb1981a4f97a92a60388efc4858d2a0d84b11e789435aa1eab903a: Status 404 returned error can't find the container with id 67d71cc4c0fb1981a4f97a92a60388efc4858d2a0d84b11e789435aa1eab903a
Sep 16 18:46:49.147525 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.147434    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-fc7857b89-4ph5b"]
Sep 16 18:46:49.150155 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.150125    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.153430 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.153406    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:46:49.153430 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.153406    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:46:49.153703 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.153483    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:46:49.153770 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.153754    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:46:49.153875 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.153776    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4vmsj\""
Sep 16 18:46:49.153996 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.153976    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:46:49.159816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.159776    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:46:49.175123 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.175084    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fc7857b89-4ph5b"]
Sep 16 18:46:49.205018 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.204979    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm" event={"ID":"81a18786-abe0-4f45-80e9-040596e64ec6","Type":"ContainerStarted","Data":"67d71cc4c0fb1981a4f97a92a60388efc4858d2a0d84b11e789435aa1eab903a"}
Sep 16 18:46:49.269802 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.269760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hndcp\" (UniqueName: \"kubernetes.io/projected/23cfabbe-7578-4b05-b064-dff6dfbeb53a-kube-api-access-hndcp\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.269997 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.269841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/23cfabbe-7578-4b05-b064-dff6dfbeb53a-telemeter-client-tls\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.269997 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.269908    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/23cfabbe-7578-4b05-b064-dff6dfbeb53a-secret-telemeter-client\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.269997 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.269926    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23cfabbe-7578-4b05-b064-dff6dfbeb53a-serving-certs-ca-bundle\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.269997 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.269945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23cfabbe-7578-4b05-b064-dff6dfbeb53a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.270241 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.270035    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23cfabbe-7578-4b05-b064-dff6dfbeb53a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.270241 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.270104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23cfabbe-7578-4b05-b064-dff6dfbeb53a-metrics-client-ca\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.270241 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.270144    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/23cfabbe-7578-4b05-b064-dff6dfbeb53a-federate-client-tls\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.371157 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.371121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/23cfabbe-7578-4b05-b064-dff6dfbeb53a-telemeter-client-tls\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.371322 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.371184    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/23cfabbe-7578-4b05-b064-dff6dfbeb53a-secret-telemeter-client\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.371322 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.371206    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23cfabbe-7578-4b05-b064-dff6dfbeb53a-serving-certs-ca-bundle\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.371322 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.371222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23cfabbe-7578-4b05-b064-dff6dfbeb53a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.371322 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.371251    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23cfabbe-7578-4b05-b064-dff6dfbeb53a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.371322 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.371287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23cfabbe-7578-4b05-b064-dff6dfbeb53a-metrics-client-ca\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.371322 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.371321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/23cfabbe-7578-4b05-b064-dff6dfbeb53a-federate-client-tls\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.371666 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.371362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hndcp\" (UniqueName: \"kubernetes.io/projected/23cfabbe-7578-4b05-b064-dff6dfbeb53a-kube-api-access-hndcp\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.372305 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.372274    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23cfabbe-7578-4b05-b064-dff6dfbeb53a-serving-certs-ca-bundle\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.372508 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.372436    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23cfabbe-7578-4b05-b064-dff6dfbeb53a-metrics-client-ca\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.372640 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.372617    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23cfabbe-7578-4b05-b064-dff6dfbeb53a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.374658 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.374612    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/23cfabbe-7578-4b05-b064-dff6dfbeb53a-secret-telemeter-client\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.374784 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.374687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/23cfabbe-7578-4b05-b064-dff6dfbeb53a-telemeter-client-tls\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.374784 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.374688    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/23cfabbe-7578-4b05-b064-dff6dfbeb53a-federate-client-tls\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.374957 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.374937    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23cfabbe-7578-4b05-b064-dff6dfbeb53a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.380106 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.380082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hndcp\" (UniqueName: \"kubernetes.io/projected/23cfabbe-7578-4b05-b064-dff6dfbeb53a-kube-api-access-hndcp\") pod \"telemeter-client-fc7857b89-4ph5b\" (UID: \"23cfabbe-7578-4b05-b064-dff6dfbeb53a\") " pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.462932 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.462893    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b"
Sep 16 18:46:49.806475 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:49.806419    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fc7857b89-4ph5b"]
Sep 16 18:46:49.810735 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:49.810707    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cfabbe_7578_4b05_b064_dff6dfbeb53a.slice/crio-fc664ae55a3bbb1a5b754ddaa2b542fa9ff1ab334788233ee4993902dbaf2ff2 WatchSource:0}: Error finding container fc664ae55a3bbb1a5b754ddaa2b542fa9ff1ab334788233ee4993902dbaf2ff2: Status 404 returned error can't find the container with id fc664ae55a3bbb1a5b754ddaa2b542fa9ff1ab334788233ee4993902dbaf2ff2
Sep 16 18:46:50.111794 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.111716    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:50.210557 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.210520    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b" event={"ID":"23cfabbe-7578-4b05-b064-dff6dfbeb53a","Type":"ContainerStarted","Data":"fc664ae55a3bbb1a5b754ddaa2b542fa9ff1ab334788233ee4993902dbaf2ff2"}
Sep 16 18:46:50.214292 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.214264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerStarted","Data":"41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe"}
Sep 16 18:46:50.214411 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.214337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerStarted","Data":"8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4"}
Sep 16 18:46:50.214411 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.214367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerStarted","Data":"ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc"}
Sep 16 18:46:50.214411 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.214398    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerStarted","Data":"3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179"}
Sep 16 18:46:50.214541 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.214434    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerStarted","Data":"a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a"}
Sep 16 18:46:50.495537 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.495505    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56f78767c5-58dv7"]
Sep 16 18:46:50.497822 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.497797    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.502105 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.502079    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:46:50.502250 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.502145    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:46:50.502250 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.502181    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:46:50.503698 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.503497    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:46:50.503698 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.503508    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:46:50.503698 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.503556    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:46:50.503698 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.503505    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l48gs\""
Sep 16 18:46:50.503698 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.503598    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:46:50.514670 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.514652    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:46:50.516006 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.515955    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f78767c5-58dv7"]
Sep 16 18:46:50.582571 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.582539    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/789924d2-54ab-4812-90a8-6cea97bab7f6-console-oauth-config\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.582741 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.582582    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbkk5\" (UniqueName: \"kubernetes.io/projected/789924d2-54ab-4812-90a8-6cea97bab7f6-kube-api-access-mbkk5\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.582741 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.582617    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-service-ca\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.582741 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.582682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-oauth-serving-cert\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.582741 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.582736    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-console-config\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.582964 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.582790    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-trusted-ca-bundle\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.582964 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.582820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/789924d2-54ab-4812-90a8-6cea97bab7f6-console-serving-cert\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.683346 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.683311    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/789924d2-54ab-4812-90a8-6cea97bab7f6-console-serving-cert\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.683523 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.683376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/789924d2-54ab-4812-90a8-6cea97bab7f6-console-oauth-config\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.685043 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.683904    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkk5\" (UniqueName: \"kubernetes.io/projected/789924d2-54ab-4812-90a8-6cea97bab7f6-kube-api-access-mbkk5\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.685043 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.683953    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-service-ca\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.685043 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.683996    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-oauth-serving-cert\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.685043 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.684108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-console-config\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.685043 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.684172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-trusted-ca-bundle\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.685043 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.684904    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-service-ca\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.685043 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.684988    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-trusted-ca-bundle\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.685731 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.685707    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-console-config\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.685731 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.685721    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-oauth-serving-cert\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.687166 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.687023    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/789924d2-54ab-4812-90a8-6cea97bab7f6-console-serving-cert\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.687166 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.687113    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/789924d2-54ab-4812-90a8-6cea97bab7f6-console-oauth-config\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.702960 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.702933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbkk5\" (UniqueName: \"kubernetes.io/projected/789924d2-54ab-4812-90a8-6cea97bab7f6-kube-api-access-mbkk5\") pod \"console-56f78767c5-58dv7\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") " pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:50.807617 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:50.807526    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:46:51.152642 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:51.152530    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f78767c5-58dv7"]
Sep 16 18:46:51.155300 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:46:51.155257    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789924d2_54ab_4812_90a8_6cea97bab7f6.slice/crio-8f43e901ae34d211edef7b5b4ee12d2a4844f08dc07bab728c16691dae20d778 WatchSource:0}: Error finding container 8f43e901ae34d211edef7b5b4ee12d2a4844f08dc07bab728c16691dae20d778: Status 404 returned error can't find the container with id 8f43e901ae34d211edef7b5b4ee12d2a4844f08dc07bab728c16691dae20d778
Sep 16 18:46:51.220659 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:51.220623    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm" event={"ID":"81a18786-abe0-4f45-80e9-040596e64ec6","Type":"ContainerStarted","Data":"e5535ab2ba415e444704e218aab9a4daf24e97dc978eae099f9301892a2c5f4c"}
Sep 16 18:46:51.224270 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:51.224239    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerStarted","Data":"f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718"}
Sep 16 18:46:51.225415 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:51.225394    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f78767c5-58dv7" event={"ID":"789924d2-54ab-4812-90a8-6cea97bab7f6","Type":"ContainerStarted","Data":"8f43e901ae34d211edef7b5b4ee12d2a4844f08dc07bab728c16691dae20d778"}
Sep 16 18:46:51.241971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:51.241924    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm" podStartSLOduration=1.6377840350000001 podStartE2EDuration="3.241908586s" totalImagesPullingTime="1.604124551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:51.241488168 +0000 UTC m=+111.044206385" watchObservedRunningTime="2026-09-16 18:46:51.241908586 +0000 UTC m=+111.044626801"
Sep 16 18:46:51.275406 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:51.275346    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.716286558 podStartE2EDuration="7.275321536s" totalImagesPullingTime="3.540963818s" totalInitContainerRuntime="18.07116ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:51.273617117 +0000 UTC m=+111.076335333" watchObservedRunningTime="2026-09-16 18:46:51.275321536 +0000 UTC m=+111.078039751"
Sep 16 18:46:52.166113 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:52.166018    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dk7qf"
Sep 16 18:46:52.232962 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:52.232925    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b" event={"ID":"23cfabbe-7578-4b05-b064-dff6dfbeb53a","Type":"ContainerStarted","Data":"54a4f937d23073727374f58226bb690fca899ed61b7717b0e677d03c8f8c772c"}
Sep 16 18:46:52.233459 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:52.232987    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b" event={"ID":"23cfabbe-7578-4b05-b064-dff6dfbeb53a","Type":"ContainerStarted","Data":"5beaaba96ae56d514349e61b5b1b374e5f2904e28a626cb7de9339461c40fb9d"}
Sep 16 18:46:52.233459 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:52.233014    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b" event={"ID":"23cfabbe-7578-4b05-b064-dff6dfbeb53a","Type":"ContainerStarted","Data":"1d8cc8f63f0afad95ca3bd545f2fe905337f226ce6b4a6a69cc1af2e19c159d6"}
Sep 16 18:46:53.197425 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:53.197389    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:53.292050 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:53.292010    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:54.243467 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:54.243433    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f78767c5-58dv7" event={"ID":"789924d2-54ab-4812-90a8-6cea97bab7f6","Type":"ContainerStarted","Data":"adcb510267aa4cfd7c5abb03fa8b7dd28ea0cf531b2090ab7554cbfb4b48db04"}
Sep 16 18:46:54.265103 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:54.265023    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-fc7857b89-4ph5b" podStartSLOduration=3.183143481 podStartE2EDuration="5.265004348s" totalImagesPullingTime="2.081860867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:52.259879315 +0000 UTC m=+112.062597521" watchObservedRunningTime="2026-09-16 18:46:54.265004348 +0000 UTC m=+114.067722563"
Sep 16 18:46:54.265263 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:54.265173    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56f78767c5-58dv7" podStartSLOduration=1.317400689 podStartE2EDuration="4.265166149s" totalImagesPullingTime="2.94776546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:54.263560622 +0000 UTC m=+114.066278839" watchObservedRunningTime="2026-09-16 18:46:54.265166149 +0000 UTC m=+114.067884380"
Sep 16 18:46:54.625863 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:54.625833    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:46:54.998770 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:54.998735    2561 patch_prober.go:28] interesting pod/image-registry-6f5dcf5979-vwfhs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:46:54.998944 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:54.998795    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs" podUID="ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:46:56.747483 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:56.747450    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:56.834663 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:56.834625    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:57.142978 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:57.142899    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f5dcf5979-vwfhs"
Sep 16 18:46:57.296694 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:57.296655    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:57.406899 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:57.406786    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:57.497581 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:57.497550    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:58.490184 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:58.490147    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f78767c5-58dv7"]
Sep 16 18:46:59.091193 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:59.091152    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.195961 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:59.195928    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.639845 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:59.639800    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv" podUID="34ca68d7-921c-4d9f-bb0f-89c482e9fb17" containerName="registry" containerID="cri-o://37bc34d793aacad72563b45b719d5d46e97b31be9c6db038c480dce123a55726" gracePeriod=30
Sep 16 18:46:59.884360 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:46:59.884331    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:47:00.071042 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.071008    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-bound-sa-token\") pod \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") "
Sep 16 18:47:00.071262 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.071049    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-trusted-ca\") pod \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") "
Sep 16 18:47:00.071262 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.071118    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-certificates\") pod \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") "
Sep 16 18:47:00.071262 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.071148    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls\") pod \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") "
Sep 16 18:47:00.071262 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.071164    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnngq\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-kube-api-access-nnngq\") pod \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") "
Sep 16 18:47:00.071262 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.071254    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-ca-trust-extracted\") pod \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") "
Sep 16 18:47:00.071534 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.071305    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-installation-pull-secrets\") pod \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") "
Sep 16 18:47:00.071534 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.071393    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-image-registry-private-configuration\") pod \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\" (UID: \"34ca68d7-921c-4d9f-bb0f-89c482e9fb17\") "
Sep 16 18:47:00.071638 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.071600    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "34ca68d7-921c-4d9f-bb0f-89c482e9fb17" (UID: "34ca68d7-921c-4d9f-bb0f-89c482e9fb17"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:00.071638 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.071618    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "34ca68d7-921c-4d9f-bb0f-89c482e9fb17" (UID: "34ca68d7-921c-4d9f-bb0f-89c482e9fb17"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:00.071736 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.071719    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-trusted-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:00.071790 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.071739    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-certificates\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:00.074023 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.073989    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "34ca68d7-921c-4d9f-bb0f-89c482e9fb17" (UID: "34ca68d7-921c-4d9f-bb0f-89c482e9fb17"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:00.074023 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.074004    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "34ca68d7-921c-4d9f-bb0f-89c482e9fb17" (UID: "34ca68d7-921c-4d9f-bb0f-89c482e9fb17"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:00.074200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.074138    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "34ca68d7-921c-4d9f-bb0f-89c482e9fb17" (UID: "34ca68d7-921c-4d9f-bb0f-89c482e9fb17"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:00.074200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.074145    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-kube-api-access-nnngq" (OuterVolumeSpecName: "kube-api-access-nnngq") pod "34ca68d7-921c-4d9f-bb0f-89c482e9fb17" (UID: "34ca68d7-921c-4d9f-bb0f-89c482e9fb17"). InnerVolumeSpecName "kube-api-access-nnngq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:00.074368 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.074348    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "34ca68d7-921c-4d9f-bb0f-89c482e9fb17" (UID: "34ca68d7-921c-4d9f-bb0f-89c482e9fb17"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:00.080366 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.080340    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "34ca68d7-921c-4d9f-bb0f-89c482e9fb17" (UID: "34ca68d7-921c-4d9f-bb0f-89c482e9fb17"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:00.172901 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.172864    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-image-registry-private-configuration\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:00.172901 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.172891    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-bound-sa-token\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:00.172901 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.172901    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-registry-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:00.172901 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.172909    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnngq\" (UniqueName: \"kubernetes.io/projected/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-kube-api-access-nnngq\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:00.173188 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.172919    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-ca-trust-extracted\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:00.173188 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.172928    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34ca68d7-921c-4d9f-bb0f-89c482e9fb17-installation-pull-secrets\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:00.268152 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.268112    2561 generic.go:356] "Generic (PLEG): container finished" podID="34ca68d7-921c-4d9f-bb0f-89c482e9fb17" containerID="37bc34d793aacad72563b45b719d5d46e97b31be9c6db038c480dce123a55726" exitCode=0
Sep 16 18:47:00.268314 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.268160    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv"
Sep 16 18:47:00.268314 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.268197    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv" event={"ID":"34ca68d7-921c-4d9f-bb0f-89c482e9fb17","Type":"ContainerDied","Data":"37bc34d793aacad72563b45b719d5d46e97b31be9c6db038c480dce123a55726"}
Sep 16 18:47:00.268314 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.268254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5884f9db8b-s8cjv" event={"ID":"34ca68d7-921c-4d9f-bb0f-89c482e9fb17","Type":"ContainerDied","Data":"9085d5a364ea4827f28d68ae4ee38503b21e94b32c49533a9e91b4e8a31800f9"}
Sep 16 18:47:00.269236 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.269215    2561 scope.go:117] "RemoveContainer" containerID="37bc34d793aacad72563b45b719d5d46e97b31be9c6db038c480dce123a55726"
Sep 16 18:47:00.278238 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.278215    2561 scope.go:117] "RemoveContainer" containerID="37bc34d793aacad72563b45b719d5d46e97b31be9c6db038c480dce123a55726"
Sep 16 18:47:00.278565 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:47:00.278543    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37bc34d793aacad72563b45b719d5d46e97b31be9c6db038c480dce123a55726\": container with ID starting with 37bc34d793aacad72563b45b719d5d46e97b31be9c6db038c480dce123a55726 not found: ID does not exist" containerID="37bc34d793aacad72563b45b719d5d46e97b31be9c6db038c480dce123a55726"
Sep 16 18:47:00.278645 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.278574    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bc34d793aacad72563b45b719d5d46e97b31be9c6db038c480dce123a55726"} err="failed to get container status \"37bc34d793aacad72563b45b719d5d46e97b31be9c6db038c480dce123a55726\": rpc error: code = NotFound desc = could not find container \"37bc34d793aacad72563b45b719d5d46e97b31be9c6db038c480dce123a55726\": container with ID starting with 37bc34d793aacad72563b45b719d5d46e97b31be9c6db038c480dce123a55726 not found: ID does not exist"
Sep 16 18:47:00.292574 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.292549    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5884f9db8b-s8cjv"]
Sep 16 18:47:00.298547 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.298524    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5884f9db8b-s8cjv"]
Sep 16 18:47:00.732946 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.732905    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34ca68d7-921c-4d9f-bb0f-89c482e9fb17" path="/var/lib/kubelet/pods/34ca68d7-921c-4d9f-bb0f-89c482e9fb17/volumes"
Sep 16 18:47:00.808316 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:00.808282    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:47:01.673152 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:01.673111    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:01.783962 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:01.783929    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:04.065123 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:04.065087    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:04.128432 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:04.128399    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:05.120208 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:05.120175    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:05.180301 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:05.180268    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:05.742756 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:05.742725    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:05.845788 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:05.845760    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:07.169417 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:07.169385    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:07.271387 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:07.271350    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:08.198106 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:08.198048    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:08.294195 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:08.294162    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:08.393801 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:08.393765    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:08.623756 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:08.623716    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:47:08.623993 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:08.623792    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:47:08.849684 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:08.849650    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:08.923760 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:08.923673    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:09.554134 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:09.554093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs\") pod \"network-metrics-daemon-65n74\" (UID: \"6b8c5f93-a582-446f-873c-5b3344f1c2c5\") " pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:47:09.556652 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:09.556616    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b8c5f93-a582-446f-873c-5b3344f1c2c5-metrics-certs\") pod \"network-metrics-daemon-65n74\" (UID: \"6b8c5f93-a582-446f-873c-5b3344f1c2c5\") " pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:47:09.641557 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:09.641518    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pq9cp\""
Sep 16 18:47:09.649801 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:09.649779    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65n74"
Sep 16 18:47:09.930403 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:09.930377    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-65n74"]
Sep 16 18:47:09.932239 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:47:09.932203    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8c5f93_a582_446f_873c_5b3344f1c2c5.slice/crio-af4d1c54b385dd4d3284cd3803df7b7cebc4688670e8111f34c60e6a5bd0579c WatchSource:0}: Error finding container af4d1c54b385dd4d3284cd3803df7b7cebc4688670e8111f34c60e6a5bd0579c: Status 404 returned error can't find the container with id af4d1c54b385dd4d3284cd3803df7b7cebc4688670e8111f34c60e6a5bd0579c
Sep 16 18:47:09.999279 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:09.999242    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:10.097931 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:10.097900    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:10.202135 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:10.202035    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:10.306883 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:10.306844    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-65n74" event={"ID":"6b8c5f93-a582-446f-873c-5b3344f1c2c5","Type":"ContainerStarted","Data":"af4d1c54b385dd4d3284cd3803df7b7cebc4688670e8111f34c60e6a5bd0579c"}
Sep 16 18:47:11.312821 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:11.312739    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-65n74" event={"ID":"6b8c5f93-a582-446f-873c-5b3344f1c2c5","Type":"ContainerStarted","Data":"853b7019b0fbcfcf5e4821f486933d8fe9374625c3aad7839471911d0fe3d687"}
Sep 16 18:47:11.312821 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:11.312796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-65n74" event={"ID":"6b8c5f93-a582-446f-873c-5b3344f1c2c5","Type":"ContainerStarted","Data":"f229caa25674900debdfbd5551d84c16ff09075246998255fd805c9549ad4bff"}
Sep 16 18:47:11.333466 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:11.333411    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-65n74" podStartSLOduration=129.297058726 podStartE2EDuration="2m10.333394206s" totalImagesPullingTime="1.03633548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:11.332164065 +0000 UTC m=+131.134882282" watchObservedRunningTime="2026-09-16 18:47:11.333394206 +0000 UTC m=+131.136112421"
Sep 16 18:47:11.512515 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:11.512481    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:11.607052 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:11.606968    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:16.298365 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:16.298329    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:16.385839 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:16.385802    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:23.510029 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.509987    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56f78767c5-58dv7" podUID="789924d2-54ab-4812-90a8-6cea97bab7f6" containerName="console" containerID="cri-o://adcb510267aa4cfd7c5abb03fa8b7dd28ea0cf531b2090ab7554cbfb4b48db04" gracePeriod=15
Sep 16 18:47:23.782106 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.782018    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:47:23.850363 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.850328    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:23.880417 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.880381    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-oauth-serving-cert\") pod \"789924d2-54ab-4812-90a8-6cea97bab7f6\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") "
Sep 16 18:47:23.880581 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.880424    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/789924d2-54ab-4812-90a8-6cea97bab7f6-console-serving-cert\") pod \"789924d2-54ab-4812-90a8-6cea97bab7f6\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") "
Sep 16 18:47:23.880581 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.880484    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-console-config\") pod \"789924d2-54ab-4812-90a8-6cea97bab7f6\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") "
Sep 16 18:47:23.880581 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.880507    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-trusted-ca-bundle\") pod \"789924d2-54ab-4812-90a8-6cea97bab7f6\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") "
Sep 16 18:47:23.880581 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.880543    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/789924d2-54ab-4812-90a8-6cea97bab7f6-console-oauth-config\") pod \"789924d2-54ab-4812-90a8-6cea97bab7f6\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") "
Sep 16 18:47:23.880782 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.880651    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-service-ca\") pod \"789924d2-54ab-4812-90a8-6cea97bab7f6\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") "
Sep 16 18:47:23.880782 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.880704    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbkk5\" (UniqueName: \"kubernetes.io/projected/789924d2-54ab-4812-90a8-6cea97bab7f6-kube-api-access-mbkk5\") pod \"789924d2-54ab-4812-90a8-6cea97bab7f6\" (UID: \"789924d2-54ab-4812-90a8-6cea97bab7f6\") "
Sep 16 18:47:23.880899 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.880864    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "789924d2-54ab-4812-90a8-6cea97bab7f6" (UID: "789924d2-54ab-4812-90a8-6cea97bab7f6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:23.880953 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.880922    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-console-config" (OuterVolumeSpecName: "console-config") pod "789924d2-54ab-4812-90a8-6cea97bab7f6" (UID: "789924d2-54ab-4812-90a8-6cea97bab7f6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:23.881046 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.881012    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "789924d2-54ab-4812-90a8-6cea97bab7f6" (UID: "789924d2-54ab-4812-90a8-6cea97bab7f6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:23.881169 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.881149    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-oauth-serving-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.881244 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.881178    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-console-config\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.881244 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.881192    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-trusted-ca-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.881244 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.881146    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-service-ca" (OuterVolumeSpecName: "service-ca") pod "789924d2-54ab-4812-90a8-6cea97bab7f6" (UID: "789924d2-54ab-4812-90a8-6cea97bab7f6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:23.882989 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.882963    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789924d2-54ab-4812-90a8-6cea97bab7f6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "789924d2-54ab-4812-90a8-6cea97bab7f6" (UID: "789924d2-54ab-4812-90a8-6cea97bab7f6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:23.883098 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.882990    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789924d2-54ab-4812-90a8-6cea97bab7f6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "789924d2-54ab-4812-90a8-6cea97bab7f6" (UID: "789924d2-54ab-4812-90a8-6cea97bab7f6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:23.883098 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.883018    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789924d2-54ab-4812-90a8-6cea97bab7f6-kube-api-access-mbkk5" (OuterVolumeSpecName: "kube-api-access-mbkk5") pod "789924d2-54ab-4812-90a8-6cea97bab7f6" (UID: "789924d2-54ab-4812-90a8-6cea97bab7f6"). InnerVolumeSpecName "kube-api-access-mbkk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:23.950720 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.950682    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:23.982486 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.982457    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/789924d2-54ab-4812-90a8-6cea97bab7f6-console-oauth-config\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.982486 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.982486    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/789924d2-54ab-4812-90a8-6cea97bab7f6-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.982663 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.982503    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbkk5\" (UniqueName: \"kubernetes.io/projected/789924d2-54ab-4812-90a8-6cea97bab7f6-kube-api-access-mbkk5\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.982663 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:23.982514    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/789924d2-54ab-4812-90a8-6cea97bab7f6-console-serving-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:47:24.365987 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:24.365952    2561 generic.go:356] "Generic (PLEG): container finished" podID="789924d2-54ab-4812-90a8-6cea97bab7f6" containerID="adcb510267aa4cfd7c5abb03fa8b7dd28ea0cf531b2090ab7554cbfb4b48db04" exitCode=0
Sep 16 18:47:24.366167 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:24.365998    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f78767c5-58dv7"
Sep 16 18:47:24.366167 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:24.366040    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f78767c5-58dv7" event={"ID":"789924d2-54ab-4812-90a8-6cea97bab7f6","Type":"ContainerDied","Data":"adcb510267aa4cfd7c5abb03fa8b7dd28ea0cf531b2090ab7554cbfb4b48db04"}
Sep 16 18:47:24.366167 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:24.366109    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f78767c5-58dv7" event={"ID":"789924d2-54ab-4812-90a8-6cea97bab7f6","Type":"ContainerDied","Data":"8f43e901ae34d211edef7b5b4ee12d2a4844f08dc07bab728c16691dae20d778"}
Sep 16 18:47:24.366167 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:24.366130    2561 scope.go:117] "RemoveContainer" containerID="adcb510267aa4cfd7c5abb03fa8b7dd28ea0cf531b2090ab7554cbfb4b48db04"
Sep 16 18:47:24.375239 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:24.375220    2561 scope.go:117] "RemoveContainer" containerID="adcb510267aa4cfd7c5abb03fa8b7dd28ea0cf531b2090ab7554cbfb4b48db04"
Sep 16 18:47:24.375504 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:47:24.375487    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcb510267aa4cfd7c5abb03fa8b7dd28ea0cf531b2090ab7554cbfb4b48db04\": container with ID starting with adcb510267aa4cfd7c5abb03fa8b7dd28ea0cf531b2090ab7554cbfb4b48db04 not found: ID does not exist" containerID="adcb510267aa4cfd7c5abb03fa8b7dd28ea0cf531b2090ab7554cbfb4b48db04"
Sep 16 18:47:24.375582 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:24.375510    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcb510267aa4cfd7c5abb03fa8b7dd28ea0cf531b2090ab7554cbfb4b48db04"} err="failed to get container status \"adcb510267aa4cfd7c5abb03fa8b7dd28ea0cf531b2090ab7554cbfb4b48db04\": rpc error: code = NotFound desc = could not find container \"adcb510267aa4cfd7c5abb03fa8b7dd28ea0cf531b2090ab7554cbfb4b48db04\": container with ID starting with adcb510267aa4cfd7c5abb03fa8b7dd28ea0cf531b2090ab7554cbfb4b48db04 not found: ID does not exist"
Sep 16 18:47:24.394674 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:24.394637    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f78767c5-58dv7"]
Sep 16 18:47:24.397638 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:24.397612    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56f78767c5-58dv7"]
Sep 16 18:47:24.733129 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:24.733096    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="789924d2-54ab-4812-90a8-6cea97bab7f6" path="/var/lib/kubelet/pods/789924d2-54ab-4812-90a8-6cea97bab7f6/volumes"
Sep 16 18:47:25.370894 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:25.370802    2561 generic.go:356] "Generic (PLEG): container finished" podID="9c177863-7189-4c95-8725-4c72172ffcf1" containerID="99a8ed150776da95723924c902e996f7079ac53c797f2e86d35664c9fb9238d4" exitCode=0
Sep 16 18:47:25.370894 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:25.370883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz" event={"ID":"9c177863-7189-4c95-8725-4c72172ffcf1","Type":"ContainerDied","Data":"99a8ed150776da95723924c902e996f7079ac53c797f2e86d35664c9fb9238d4"}
Sep 16 18:47:25.371267 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:25.371248    2561 scope.go:117] "RemoveContainer" containerID="99a8ed150776da95723924c902e996f7079ac53c797f2e86d35664c9fb9238d4"
Sep 16 18:47:26.377646 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:26.377609    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-p8frz" event={"ID":"9c177863-7189-4c95-8725-4c72172ffcf1","Type":"ContainerStarted","Data":"19ef9d9df07b7e63c2eab458f2316b227d611a8bbd56d2ccd111dcdf7b677b8c"}
Sep 16 18:47:28.628771 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:28.628742    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:47:28.632734 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:28.632706    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-69566c86c6-gt6sm"
Sep 16 18:47:31.398258 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:31.398218    2561 generic.go:356] "Generic (PLEG): container finished" podID="84e3975d-dbd7-4c94-a16c-e61da230a8c7" containerID="3d5a3f61b4d0925a4302e43d3e4c144e0c9983133244beb4d660f6c94b45d717" exitCode=0
Sep 16 18:47:31.398646 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:31.398293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-f2clx" event={"ID":"84e3975d-dbd7-4c94-a16c-e61da230a8c7","Type":"ContainerDied","Data":"3d5a3f61b4d0925a4302e43d3e4c144e0c9983133244beb4d660f6c94b45d717"}
Sep 16 18:47:31.398646 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:31.398621    2561 scope.go:117] "RemoveContainer" containerID="3d5a3f61b4d0925a4302e43d3e4c144e0c9983133244beb4d660f6c94b45d717"
Sep 16 18:47:32.404258 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:32.404218    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-f2clx" event={"ID":"84e3975d-dbd7-4c94-a16c-e61da230a8c7","Type":"ContainerStarted","Data":"737ba091a41f6bf0a34f31f078ae591ba6e92c109e1a24d16618703e567ed69d"}
Sep 16 18:47:38.864637 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:38.864601    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:38.954120 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:38.954085    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:44.994293 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:44.994249    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:45.129429 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:45.129315    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:58.054364 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:47:58.054331    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:00.049719 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:00.049660    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:00.180166 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:00.180135    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:04.480040 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:04.480008    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:04.480479 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:04.480315    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="alertmanager" containerID="cri-o://a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a" gracePeriod=120
Sep 16 18:48:04.480479 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:04.480395    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="kube-rbac-proxy-metric" containerID="cri-o://41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe" gracePeriod=120
Sep 16 18:48:04.480479 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:04.480400    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="kube-rbac-proxy-web" containerID="cri-o://ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc" gracePeriod=120
Sep 16 18:48:04.480479 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:04.480436    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="kube-rbac-proxy" containerID="cri-o://8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4" gracePeriod=120
Sep 16 18:48:04.480697 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:04.480504    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="prom-label-proxy" containerID="cri-o://f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718" gracePeriod=120
Sep 16 18:48:04.480697 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:04.480432    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="config-reloader" containerID="cri-o://3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179" gracePeriod=120
Sep 16 18:48:05.547744 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.547715    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerID="f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718" exitCode=0
Sep 16 18:48:05.547744 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.547740    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerID="8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4" exitCode=0
Sep 16 18:48:05.547744 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.547747    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerID="3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179" exitCode=0
Sep 16 18:48:05.548226 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.547753    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerID="a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a" exitCode=0
Sep 16 18:48:05.548226 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.547784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerDied","Data":"f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718"}
Sep 16 18:48:05.548226 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.547850    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerDied","Data":"8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4"}
Sep 16 18:48:05.548226 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.547890    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerDied","Data":"3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179"}
Sep 16 18:48:05.548226 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.547918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerDied","Data":"a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a"}
Sep 16 18:48:05.730209 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.730180    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:05.837561 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.837472    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-tls-assets\") pod \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") "
Sep 16 18:48:05.837561 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.837509    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy-web\") pod \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") "
Sep 16 18:48:05.837561 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.837549    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-main-tls\") pod \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") "
Sep 16 18:48:05.837807 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.837587    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy\") pod \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") "
Sep 16 18:48:05.837807 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.837615    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-web-config\") pod \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") "
Sep 16 18:48:05.837807 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.837733    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-alertmanager-trusted-ca-bundle\") pod \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") "
Sep 16 18:48:05.837807 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.837787    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") "
Sep 16 18:48:05.838013 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.837838    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59fdl\" (UniqueName: \"kubernetes.io/projected/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-kube-api-access-59fdl\") pod \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") "
Sep 16 18:48:05.838013 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.837868    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-config-volume\") pod \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") "
Sep 16 18:48:05.838013 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.837902    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-alertmanager-main-db\") pod \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") "
Sep 16 18:48:05.838013 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.837955    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-config-out\") pod \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") "
Sep 16 18:48:05.838013 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.837991    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-cluster-tls-config\") pod \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") "
Sep 16 18:48:05.838262 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.838021    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-metrics-client-ca\") pod \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\" (UID: \"2b030fbd-db7e-4c90-94e0-83ceb26dbd93\") "
Sep 16 18:48:05.838694 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.838664    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "2b030fbd-db7e-4c90-94e0-83ceb26dbd93" (UID: "2b030fbd-db7e-4c90-94e0-83ceb26dbd93"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:05.838820 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.838661    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "2b030fbd-db7e-4c90-94e0-83ceb26dbd93" (UID: "2b030fbd-db7e-4c90-94e0-83ceb26dbd93"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:05.841630 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.841315    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "2b030fbd-db7e-4c90-94e0-83ceb26dbd93" (UID: "2b030fbd-db7e-4c90-94e0-83ceb26dbd93"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:05.841751 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.841701    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "2b030fbd-db7e-4c90-94e0-83ceb26dbd93" (UID: "2b030fbd-db7e-4c90-94e0-83ceb26dbd93"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:05.842228 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.842190    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2b030fbd-db7e-4c90-94e0-83ceb26dbd93" (UID: "2b030fbd-db7e-4c90-94e0-83ceb26dbd93"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:05.842336 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.842261    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "2b030fbd-db7e-4c90-94e0-83ceb26dbd93" (UID: "2b030fbd-db7e-4c90-94e0-83ceb26dbd93"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:05.842492 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.842463    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "2b030fbd-db7e-4c90-94e0-83ceb26dbd93" (UID: "2b030fbd-db7e-4c90-94e0-83ceb26dbd93"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:05.842492 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.842482    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "2b030fbd-db7e-4c90-94e0-83ceb26dbd93" (UID: "2b030fbd-db7e-4c90-94e0-83ceb26dbd93"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:05.842660 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.842604    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-config-volume" (OuterVolumeSpecName: "config-volume") pod "2b030fbd-db7e-4c90-94e0-83ceb26dbd93" (UID: "2b030fbd-db7e-4c90-94e0-83ceb26dbd93"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:05.843279 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.843253    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-kube-api-access-59fdl" (OuterVolumeSpecName: "kube-api-access-59fdl") pod "2b030fbd-db7e-4c90-94e0-83ceb26dbd93" (UID: "2b030fbd-db7e-4c90-94e0-83ceb26dbd93"). InnerVolumeSpecName "kube-api-access-59fdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:05.843720 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.843697    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-config-out" (OuterVolumeSpecName: "config-out") pod "2b030fbd-db7e-4c90-94e0-83ceb26dbd93" (UID: "2b030fbd-db7e-4c90-94e0-83ceb26dbd93"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:05.846948 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.846923    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "2b030fbd-db7e-4c90-94e0-83ceb26dbd93" (UID: "2b030fbd-db7e-4c90-94e0-83ceb26dbd93"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:05.853614 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.853588    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-web-config" (OuterVolumeSpecName: "web-config") pod "2b030fbd-db7e-4c90-94e0-83ceb26dbd93" (UID: "2b030fbd-db7e-4c90-94e0-83ceb26dbd93"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:05.939195 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.939159    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:48:05.939195 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.939193    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59fdl\" (UniqueName: \"kubernetes.io/projected/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-kube-api-access-59fdl\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:48:05.939403 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.939204    2561 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-config-volume\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:48:05.939403 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.939214    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-alertmanager-main-db\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:48:05.939403 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.939223    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-config-out\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:48:05.939403 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.939231    2561 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-cluster-tls-config\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:48:05.939403 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.939239    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-metrics-client-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:48:05.939403 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.939250    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-tls-assets\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:48:05.939403 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.939260    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:48:05.939403 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.939268    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-main-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:48:05.939403 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.939277    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:48:05.939403 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.939285    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-web-config\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:48:05.939403 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:05.939293    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b030fbd-db7e-4c90-94e0-83ceb26dbd93-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:48:06.558222 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.558192    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerID="41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe" exitCode=0
Sep 16 18:48:06.558222 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.558218    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerID="ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc" exitCode=0
Sep 16 18:48:06.558565 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.558284    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.558565 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.558275    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerDied","Data":"41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe"}
Sep 16 18:48:06.558565 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.558436    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerDied","Data":"ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc"}
Sep 16 18:48:06.558565 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.558484    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b030fbd-db7e-4c90-94e0-83ceb26dbd93","Type":"ContainerDied","Data":"675b290756526a72fba7f8f6c616a21225fc6ac83d65c16dc67da786a9f675c6"}
Sep 16 18:48:06.558565 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.558506    2561 scope.go:117] "RemoveContainer" containerID="f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718"
Sep 16 18:48:06.566577 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.566554    2561 scope.go:117] "RemoveContainer" containerID="41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe"
Sep 16 18:48:06.574139 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.574117    2561 scope.go:117] "RemoveContainer" containerID="8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4"
Sep 16 18:48:06.581941 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.581918    2561 scope.go:117] "RemoveContainer" containerID="ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc"
Sep 16 18:48:06.589817 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.589796    2561 scope.go:117] "RemoveContainer" containerID="3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179"
Sep 16 18:48:06.597704 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.597631    2561 scope.go:117] "RemoveContainer" containerID="a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a"
Sep 16 18:48:06.599877 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.599857    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:06.600975 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.600955    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:06.606374 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.606357    2561 scope.go:117] "RemoveContainer" containerID="464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d"
Sep 16 18:48:06.613582 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.613558    2561 scope.go:117] "RemoveContainer" containerID="f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718"
Sep 16 18:48:06.613873 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:48:06.613855    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718\": container with ID starting with f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718 not found: ID does not exist" containerID="f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718"
Sep 16 18:48:06.613927 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.613882    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718"} err="failed to get container status \"f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718\": rpc error: code = NotFound desc = could not find container \"f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718\": container with ID starting with f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718 not found: ID does not exist"
Sep 16 18:48:06.613927 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.613902    2561 scope.go:117] "RemoveContainer" containerID="41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe"
Sep 16 18:48:06.614189 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:48:06.614170    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe\": container with ID starting with 41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe not found: ID does not exist" containerID="41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe"
Sep 16 18:48:06.614264 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.614199    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe"} err="failed to get container status \"41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe\": rpc error: code = NotFound desc = could not find container \"41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe\": container with ID starting with 41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe not found: ID does not exist"
Sep 16 18:48:06.614264 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.614224    2561 scope.go:117] "RemoveContainer" containerID="8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4"
Sep 16 18:48:06.614481 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:48:06.614459    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4\": container with ID starting with 8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4 not found: ID does not exist" containerID="8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4"
Sep 16 18:48:06.614578 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.614483    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4"} err="failed to get container status \"8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4\": rpc error: code = NotFound desc = could not find container \"8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4\": container with ID starting with 8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4 not found: ID does not exist"
Sep 16 18:48:06.614578 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.614498    2561 scope.go:117] "RemoveContainer" containerID="ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc"
Sep 16 18:48:06.614762 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:48:06.614747    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc\": container with ID starting with ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc not found: ID does not exist" containerID="ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc"
Sep 16 18:48:06.614805 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.614766    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc"} err="failed to get container status \"ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc\": rpc error: code = NotFound desc = could not find container \"ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc\": container with ID starting with ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc not found: ID does not exist"
Sep 16 18:48:06.614805 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.614779    2561 scope.go:117] "RemoveContainer" containerID="3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179"
Sep 16 18:48:06.615005 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:48:06.614986    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179\": container with ID starting with 3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179 not found: ID does not exist" containerID="3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179"
Sep 16 18:48:06.615119 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.615004    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179"} err="failed to get container status \"3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179\": rpc error: code = NotFound desc = could not find container \"3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179\": container with ID starting with 3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179 not found: ID does not exist"
Sep 16 18:48:06.615119 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.615016    2561 scope.go:117] "RemoveContainer" containerID="a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a"
Sep 16 18:48:06.615282 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:48:06.615267    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a\": container with ID starting with a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a not found: ID does not exist" containerID="a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a"
Sep 16 18:48:06.615328 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.615287    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a"} err="failed to get container status \"a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a\": rpc error: code = NotFound desc = could not find container \"a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a\": container with ID starting with a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a not found: ID does not exist"
Sep 16 18:48:06.615328 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.615301    2561 scope.go:117] "RemoveContainer" containerID="464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d"
Sep 16 18:48:06.615530 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:48:06.615514    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d\": container with ID starting with 464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d not found: ID does not exist" containerID="464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d"
Sep 16 18:48:06.615574 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.615536    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d"} err="failed to get container status \"464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d\": rpc error: code = NotFound desc = could not find container \"464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d\": container with ID starting with 464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d not found: ID does not exist"
Sep 16 18:48:06.615574 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.615548    2561 scope.go:117] "RemoveContainer" containerID="f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718"
Sep 16 18:48:06.615763 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.615747    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718"} err="failed to get container status \"f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718\": rpc error: code = NotFound desc = could not find container \"f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718\": container with ID starting with f7b43a468d6005007d2db43982a866b6d3b4a445fc53b476edd22f5595fdb718 not found: ID does not exist"
Sep 16 18:48:06.615811 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.615763    2561 scope.go:117] "RemoveContainer" containerID="41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe"
Sep 16 18:48:06.615975 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.615957    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe"} err="failed to get container status \"41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe\": rpc error: code = NotFound desc = could not find container \"41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe\": container with ID starting with 41aca08daddd89aae043bbb23d1f75889ab7b22e0d78fdc9dde596ed6f029fbe not found: ID does not exist"
Sep 16 18:48:06.616019 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.615977    2561 scope.go:117] "RemoveContainer" containerID="8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4"
Sep 16 18:48:06.616272 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.616253    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4"} err="failed to get container status \"8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4\": rpc error: code = NotFound desc = could not find container \"8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4\": container with ID starting with 8b3b16ba5b2a66920b4b9e56491e05f0d38d17441fcb3e3fbf5b42565c3060a4 not found: ID does not exist"
Sep 16 18:48:06.616322 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.616273    2561 scope.go:117] "RemoveContainer" containerID="ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc"
Sep 16 18:48:06.616481 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.616461    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc"} err="failed to get container status \"ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc\": rpc error: code = NotFound desc = could not find container \"ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc\": container with ID starting with ec68943c24f1cb8bd434bd66afb6c6c28eea23fef5a5466dc4527eb03b0422bc not found: ID does not exist"
Sep 16 18:48:06.616523 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.616483    2561 scope.go:117] "RemoveContainer" containerID="3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179"
Sep 16 18:48:06.616684 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.616668    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179"} err="failed to get container status \"3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179\": rpc error: code = NotFound desc = could not find container \"3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179\": container with ID starting with 3523385806e25d056c51d379c28557c464ca0c9fca47ed462dfa89303c7dd179 not found: ID does not exist"
Sep 16 18:48:06.616727 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.616684    2561 scope.go:117] "RemoveContainer" containerID="a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a"
Sep 16 18:48:06.616901 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.616877    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a"} err="failed to get container status \"a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a\": rpc error: code = NotFound desc = could not find container \"a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a\": container with ID starting with a522b69151c85ac06d3c5f51b4c59259fd952cd90f9e839db1adae473ed4d31a not found: ID does not exist"
Sep 16 18:48:06.616943 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.616901    2561 scope.go:117] "RemoveContainer" containerID="464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d"
Sep 16 18:48:06.617152 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.617133    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d"} err="failed to get container status \"464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d\": rpc error: code = NotFound desc = could not find container \"464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d\": container with ID starting with 464dd49400fd80038672b41bc4fe4778c12e2facdff36a02f7327c62be2d183d not found: ID does not exist"
Sep 16 18:48:06.673795 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.673756    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:06.674088 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674050    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="kube-rbac-proxy"
Sep 16 18:48:06.674088 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674081    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="kube-rbac-proxy"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674096    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="init-config-reloader"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674103    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="init-config-reloader"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674110    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="kube-rbac-proxy-metric"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674115    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="kube-rbac-proxy-metric"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674124    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="prom-label-proxy"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674130    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="prom-label-proxy"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674141    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="config-reloader"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674146    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="config-reloader"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674156    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="alertmanager"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674162    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="alertmanager"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674175    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="kube-rbac-proxy-web"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674181    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="kube-rbac-proxy-web"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674188    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="789924d2-54ab-4812-90a8-6cea97bab7f6" containerName="console"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674193    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="789924d2-54ab-4812-90a8-6cea97bab7f6" containerName="console"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674199    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34ca68d7-921c-4d9f-bb0f-89c482e9fb17" containerName="registry"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674203    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="34ca68d7-921c-4d9f-bb0f-89c482e9fb17" containerName="registry"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674251    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="34ca68d7-921c-4d9f-bb0f-89c482e9fb17" containerName="registry"
Sep 16 18:48:06.674254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674261    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="kube-rbac-proxy-web"
Sep 16 18:48:06.675019 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674268    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="kube-rbac-proxy"
Sep 16 18:48:06.675019 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674274    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="prom-label-proxy"
Sep 16 18:48:06.675019 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674279    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="789924d2-54ab-4812-90a8-6cea97bab7f6" containerName="console"
Sep 16 18:48:06.675019 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674285    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="kube-rbac-proxy-metric"
Sep 16 18:48:06.675019 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674290    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="alertmanager"
Sep 16 18:48:06.675019 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.674295    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" containerName="config-reloader"
Sep 16 18:48:06.677987 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.677969    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.701408 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.701374    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:48:06.701586 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.701415    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:48:06.701586 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.701494    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:48:06.701586 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.701565    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:48:06.701716 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.701688    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6thqs\""
Sep 16 18:48:06.706822 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.706787    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:48:06.707002 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.706845    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:48:06.707002 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.706901    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:48:06.707002 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.706923    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:48:06.722379 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.722345    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:48:06.733184 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.733152    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b030fbd-db7e-4c90-94e0-83ceb26dbd93" path="/var/lib/kubelet/pods/2b030fbd-db7e-4c90-94e0-83ceb26dbd93/volumes"
Sep 16 18:48:06.745673 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.745632    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-web-config\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.745825 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.745674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.745825 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.745706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.745825 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.745749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.745825 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.745779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.745956 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.745838    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.745956 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.745879    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.745956 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.745903    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hll2s\" (UniqueName: \"kubernetes.io/projected/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-kube-api-access-hll2s\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.745956 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.745927    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-config-out\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.745956 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.745942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.746146 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.745959    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.746146 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.745980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-config-volume\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.746146 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.746002    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.826859 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.826769    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:06.847499 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.847457    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.847680 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.847545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.847680 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.847626    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.847680 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.847655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hll2s\" (UniqueName: \"kubernetes.io/projected/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-kube-api-access-hll2s\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.847889 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.847685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-config-out\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.847889 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.847714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.847889 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.847742    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.847889 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.847773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-config-volume\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.847889 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.847798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.848162 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.847984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-web-config\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.848162 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.848021    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.848162 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.848095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.848162 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.848157    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.848346 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.848175    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.849209 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.849152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.849550 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.849499    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.850831 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.850803    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.851739 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.851517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-web-config\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.851988 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.851964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.852117 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.851971    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.852265 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.852241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.852332 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.852291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-config-out\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.852332 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.852319    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-config-volume\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.852502 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.852458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.853328 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.853312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.857697 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.857672    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hll2s\" (UniqueName: \"kubernetes.io/projected/bf491168-8cb8-4f07-81a0-d2c7a9ad0d90-kube-api-access-hll2s\") pod \"alertmanager-main-0\" (UID: \"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:06.987938 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:06.987890    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:07.290960 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:07.290936    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:07.292123 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:48:07.292100    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf491168_8cb8_4f07_81a0_d2c7a9ad0d90.slice/crio-4921c546e2fb6fd0c179a4873809bf572e27482a6f8a532de379247e470ecd10 WatchSource:0}: Error finding container 4921c546e2fb6fd0c179a4873809bf572e27482a6f8a532de379247e470ecd10: Status 404 returned error can't find the container with id 4921c546e2fb6fd0c179a4873809bf572e27482a6f8a532de379247e470ecd10
Sep 16 18:48:07.564380 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:07.564292    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf491168-8cb8-4f07-81a0-d2c7a9ad0d90" containerID="e475259bacab2ed8babbf6241a339564bbea986f8b1265a5a27387575feae1ad" exitCode=0
Sep 16 18:48:07.564380 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:07.564352    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90","Type":"ContainerDied","Data":"e475259bacab2ed8babbf6241a339564bbea986f8b1265a5a27387575feae1ad"}
Sep 16 18:48:07.564765 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:07.564403    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90","Type":"ContainerStarted","Data":"4921c546e2fb6fd0c179a4873809bf572e27482a6f8a532de379247e470ecd10"}
Sep 16 18:48:08.571589 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:08.571551    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90","Type":"ContainerStarted","Data":"7b8fff045c908910c722477127b29e05833fe28e4f0070de34122a2c02a689ac"}
Sep 16 18:48:08.571944 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:08.571631    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90","Type":"ContainerStarted","Data":"451b114bf27f82e4a82680a2ec7eeabf042a94a3fc556fb50e74ba8d09efaa31"}
Sep 16 18:48:08.571944 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:08.571672    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90","Type":"ContainerStarted","Data":"365bb221755b72b771fa0a845097a1f517d04ba6c8f17b6561892061a1652e82"}
Sep 16 18:48:08.571944 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:08.571701    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90","Type":"ContainerStarted","Data":"0708c24716a908fb497883474c09aa4a5d81371c50b6fcde7fc43950a6ad7d99"}
Sep 16 18:48:08.571944 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:08.571724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90","Type":"ContainerStarted","Data":"b7536b085e73df96e0b01c56c09d5ee102db75035a0e3149a5204ba8667a44af"}
Sep 16 18:48:08.571944 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:08.571764    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf491168-8cb8-4f07-81a0-d2c7a9ad0d90","Type":"ContainerStarted","Data":"381062833cf66bce276beda301f1a0ad2d73f4627d9843ec490e12986e85fe7c"}
Sep 16 18:48:08.605520 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:08.605456    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.588547557 podStartE2EDuration="2.605433093s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.885536ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:08.602727662 +0000 UTC m=+188.405445888" watchObservedRunningTime="2026-09-16 18:48:08.605433093 +0000 UTC m=+188.408151309"
Sep 16 18:48:11.991503 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:11.991469    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:12.080034 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:12.079973    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:23.863023 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:23.862987    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:23.962864 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:23.962833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:25.254903 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.254871    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b45b774d6-wz2fk"]
Sep 16 18:48:25.257925 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.257904    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.261711 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.261683    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:48:25.261841 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.261689    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:48:25.262463 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.262443    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:48:25.262463 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.262460    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:48:25.262652 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.262443    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l48gs\""
Sep 16 18:48:25.262652 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.262487    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:48:25.263215 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.263199    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:48:25.263792 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.263772    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:48:25.269017 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.268995    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:48:25.270164 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.270146    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b45b774d6-wz2fk"]
Sep 16 18:48:25.331139 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.331089    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:25.406517 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.406484    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-oauth-serving-cert\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.406683 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.406527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e622d11-f99f-43d8-af28-88e091c685e3-console-serving-cert\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.406683 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.406557    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-console-config\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.406683 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.406620    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e622d11-f99f-43d8-af28-88e091c685e3-console-oauth-config\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.406683 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.406660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvg5v\" (UniqueName: \"kubernetes.io/projected/8e622d11-f99f-43d8-af28-88e091c685e3-kube-api-access-fvg5v\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.406815 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.406690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-service-ca\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.406815 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.406727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-trusted-ca-bundle\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.427573 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.427540    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:25.507926 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.507824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-oauth-serving-cert\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.507926 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.507875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e622d11-f99f-43d8-af28-88e091c685e3-console-serving-cert\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.508203 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.507990    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-console-config\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.508203 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.508045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e622d11-f99f-43d8-af28-88e091c685e3-console-oauth-config\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.508203 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.508102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvg5v\" (UniqueName: \"kubernetes.io/projected/8e622d11-f99f-43d8-af28-88e091c685e3-kube-api-access-fvg5v\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.508203 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.508138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-service-ca\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.508203 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.508197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-trusted-ca-bundle\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.508671 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.508648    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-oauth-serving-cert\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.508956 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.508934    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-console-config\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.509311 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.509289    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-trusted-ca-bundle\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.509537 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.509515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-service-ca\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.510728 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.510705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e622d11-f99f-43d8-af28-88e091c685e3-console-serving-cert\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.510847 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.510824    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e622d11-f99f-43d8-af28-88e091c685e3-console-oauth-config\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.517399 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.517375    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvg5v\" (UniqueName: \"kubernetes.io/projected/8e622d11-f99f-43d8-af28-88e091c685e3-kube-api-access-fvg5v\") pod \"console-6b45b774d6-wz2fk\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") " pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.569075 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.569005    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:25.844941 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:25.844916    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b45b774d6-wz2fk"]
Sep 16 18:48:25.847705 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:48:25.847678    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e622d11_f99f_43d8_af28_88e091c685e3.slice/crio-964b7658953475858d0fc3c59338ee37ab000d359ab740596db9bec4a8a8f64e WatchSource:0}: Error finding container 964b7658953475858d0fc3c59338ee37ab000d359ab740596db9bec4a8a8f64e: Status 404 returned error can't find the container with id 964b7658953475858d0fc3c59338ee37ab000d359ab740596db9bec4a8a8f64e
Sep 16 18:48:26.646701 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:26.646663    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b45b774d6-wz2fk" event={"ID":"8e622d11-f99f-43d8-af28-88e091c685e3","Type":"ContainerStarted","Data":"5e5c38db2046783f7cce56150cb43176ec02a8cdaeb0657cc4c428f1b178449c"}
Sep 16 18:48:26.647093 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:26.646715    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b45b774d6-wz2fk" event={"ID":"8e622d11-f99f-43d8-af28-88e091c685e3","Type":"ContainerStarted","Data":"964b7658953475858d0fc3c59338ee37ab000d359ab740596db9bec4a8a8f64e"}
Sep 16 18:48:26.667425 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:26.667375    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b45b774d6-wz2fk" podStartSLOduration=1.667358792 podStartE2EDuration="1.667358792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:26.66729959 +0000 UTC m=+206.470017806" watchObservedRunningTime="2026-09-16 18:48:26.667358792 +0000 UTC m=+206.470077008"
Sep 16 18:48:35.569334 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:35.569286    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:35.569819 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:35.569380    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:35.574253 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:35.574230    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:35.684072 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:35.684026    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:48:35.875307 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:35.875219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:35.978291 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:35.978250    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:36.076163 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:48:36.076132    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:02.182032 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:02.181995    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:02.282735 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:02.282704    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:16.692135 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:16.692095    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:16.792532 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:16.792499    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:16.891872 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:16.891837    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:17.809401 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:17.809366    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"]
Sep 16 18:49:17.814255 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:17.814232    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"
Sep 16 18:49:17.817374 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:17.817349    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:49:17.817509 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:17.817408    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:49:17.817509 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:17.817408    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v2xds\""
Sep 16 18:49:17.823438 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:17.823416    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"]
Sep 16 18:49:17.940668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:17.940630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ba67f78-6c99-4e78-ba87-529e861d135d-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2\" (UID: \"1ba67f78-6c99-4e78-ba87-529e861d135d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"
Sep 16 18:49:17.940856 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:17.940701    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl4bh\" (UniqueName: \"kubernetes.io/projected/1ba67f78-6c99-4e78-ba87-529e861d135d-kube-api-access-bl4bh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2\" (UID: \"1ba67f78-6c99-4e78-ba87-529e861d135d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"
Sep 16 18:49:17.940856 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:17.940762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ba67f78-6c99-4e78-ba87-529e861d135d-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2\" (UID: \"1ba67f78-6c99-4e78-ba87-529e861d135d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"
Sep 16 18:49:18.041971 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:18.041935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ba67f78-6c99-4e78-ba87-529e861d135d-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2\" (UID: \"1ba67f78-6c99-4e78-ba87-529e861d135d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"
Sep 16 18:49:18.042185 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:18.042001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ba67f78-6c99-4e78-ba87-529e861d135d-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2\" (UID: \"1ba67f78-6c99-4e78-ba87-529e861d135d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"
Sep 16 18:49:18.042185 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:18.042093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl4bh\" (UniqueName: \"kubernetes.io/projected/1ba67f78-6c99-4e78-ba87-529e861d135d-kube-api-access-bl4bh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2\" (UID: \"1ba67f78-6c99-4e78-ba87-529e861d135d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"
Sep 16 18:49:18.042472 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:18.042448    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ba67f78-6c99-4e78-ba87-529e861d135d-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2\" (UID: \"1ba67f78-6c99-4e78-ba87-529e861d135d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"
Sep 16 18:49:18.042543 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:18.042450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ba67f78-6c99-4e78-ba87-529e861d135d-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2\" (UID: \"1ba67f78-6c99-4e78-ba87-529e861d135d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"
Sep 16 18:49:18.051973 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:18.051947    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl4bh\" (UniqueName: \"kubernetes.io/projected/1ba67f78-6c99-4e78-ba87-529e861d135d-kube-api-access-bl4bh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2\" (UID: \"1ba67f78-6c99-4e78-ba87-529e861d135d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"
Sep 16 18:49:18.123997 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:18.123896    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"
Sep 16 18:49:18.392758 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:18.392648    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"]
Sep 16 18:49:18.395726 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:49:18.395681    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba67f78_6c99_4e78_ba87_529e861d135d.slice/crio-834d5b0d25b524d0ed5223b04296bc5f216810814ed7956c671b4ec615aeebb8 WatchSource:0}: Error finding container 834d5b0d25b524d0ed5223b04296bc5f216810814ed7956c671b4ec615aeebb8: Status 404 returned error can't find the container with id 834d5b0d25b524d0ed5223b04296bc5f216810814ed7956c671b4ec615aeebb8
Sep 16 18:49:18.848762 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:18.848728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2" event={"ID":"1ba67f78-6c99-4e78-ba87-529e861d135d","Type":"ContainerStarted","Data":"834d5b0d25b524d0ed5223b04296bc5f216810814ed7956c671b4ec615aeebb8"}
Sep 16 18:49:24.876811 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:24.876776    2561 generic.go:356] "Generic (PLEG): container finished" podID="1ba67f78-6c99-4e78-ba87-529e861d135d" containerID="49eca6d828ade70b0f4617584cae9dab77a4df3ae6a22b9d7ce0d98b09fe7d37" exitCode=0
Sep 16 18:49:24.877204 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:24.876876    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2" event={"ID":"1ba67f78-6c99-4e78-ba87-529e861d135d","Type":"ContainerDied","Data":"49eca6d828ade70b0f4617584cae9dab77a4df3ae6a22b9d7ce0d98b09fe7d37"}
Sep 16 18:49:26.887173 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:26.887142    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2" event={"ID":"1ba67f78-6c99-4e78-ba87-529e861d135d","Type":"ContainerStarted","Data":"80a06cda83980ee2bd7af321460a3b8eb1abf6eb2d58b74775891df8ac79bc62"}
Sep 16 18:49:27.893275 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:27.893234    2561 generic.go:356] "Generic (PLEG): container finished" podID="1ba67f78-6c99-4e78-ba87-529e861d135d" containerID="80a06cda83980ee2bd7af321460a3b8eb1abf6eb2d58b74775891df8ac79bc62" exitCode=0
Sep 16 18:49:27.893667 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:27.893286    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2" event={"ID":"1ba67f78-6c99-4e78-ba87-529e861d135d","Type":"ContainerDied","Data":"80a06cda83980ee2bd7af321460a3b8eb1abf6eb2d58b74775891df8ac79bc62"}
Sep 16 18:49:34.929254 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:34.929216    2561 generic.go:356] "Generic (PLEG): container finished" podID="1ba67f78-6c99-4e78-ba87-529e861d135d" containerID="626e39da1708b0142e8a60872737fc699da83ace1c9665ac1b1a3fe1d315ba9f" exitCode=0
Sep 16 18:49:34.929663 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:34.929269    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2" event={"ID":"1ba67f78-6c99-4e78-ba87-529e861d135d","Type":"ContainerDied","Data":"626e39da1708b0142e8a60872737fc699da83ace1c9665ac1b1a3fe1d315ba9f"}
Sep 16 18:49:35.603761 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:35.603729    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:35.684222 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:35.684192    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:36.057025 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:36.056997    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"
Sep 16 18:49:36.213412 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:36.213387    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ba67f78-6c99-4e78-ba87-529e861d135d-util\") pod \"1ba67f78-6c99-4e78-ba87-529e861d135d\" (UID: \"1ba67f78-6c99-4e78-ba87-529e861d135d\") "
Sep 16 18:49:36.213574 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:36.213459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl4bh\" (UniqueName: \"kubernetes.io/projected/1ba67f78-6c99-4e78-ba87-529e861d135d-kube-api-access-bl4bh\") pod \"1ba67f78-6c99-4e78-ba87-529e861d135d\" (UID: \"1ba67f78-6c99-4e78-ba87-529e861d135d\") "
Sep 16 18:49:36.213574 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:36.213479    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ba67f78-6c99-4e78-ba87-529e861d135d-bundle\") pod \"1ba67f78-6c99-4e78-ba87-529e861d135d\" (UID: \"1ba67f78-6c99-4e78-ba87-529e861d135d\") "
Sep 16 18:49:36.214263 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:36.214158    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba67f78-6c99-4e78-ba87-529e861d135d-bundle" (OuterVolumeSpecName: "bundle") pod "1ba67f78-6c99-4e78-ba87-529e861d135d" (UID: "1ba67f78-6c99-4e78-ba87-529e861d135d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:36.215855 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:36.215829    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba67f78-6c99-4e78-ba87-529e861d135d-kube-api-access-bl4bh" (OuterVolumeSpecName: "kube-api-access-bl4bh") pod "1ba67f78-6c99-4e78-ba87-529e861d135d" (UID: "1ba67f78-6c99-4e78-ba87-529e861d135d"). InnerVolumeSpecName "kube-api-access-bl4bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:36.217477 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:36.217452    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba67f78-6c99-4e78-ba87-529e861d135d-util" (OuterVolumeSpecName: "util") pod "1ba67f78-6c99-4e78-ba87-529e861d135d" (UID: "1ba67f78-6c99-4e78-ba87-529e861d135d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:36.314460 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:36.314414    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ba67f78-6c99-4e78-ba87-529e861d135d-util\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:49:36.314460 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:36.314456    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl4bh\" (UniqueName: \"kubernetes.io/projected/1ba67f78-6c99-4e78-ba87-529e861d135d-kube-api-access-bl4bh\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:49:36.314460 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:36.314467    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ba67f78-6c99-4e78-ba87-529e861d135d-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:49:36.944406 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:36.944379    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2"
Sep 16 18:49:36.944570 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:36.944371    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qjr2" event={"ID":"1ba67f78-6c99-4e78-ba87-529e861d135d","Type":"ContainerDied","Data":"834d5b0d25b524d0ed5223b04296bc5f216810814ed7956c671b4ec615aeebb8"}
Sep 16 18:49:36.944570 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:36.944527    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="834d5b0d25b524d0ed5223b04296bc5f216810814ed7956c671b4ec615aeebb8"
Sep 16 18:49:40.259569 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:40.259517    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:40.472584 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:40.472547    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:42.930429 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.930389    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss"]
Sep 16 18:49:42.930898 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.930848    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ba67f78-6c99-4e78-ba87-529e861d135d" containerName="util"
Sep 16 18:49:42.930898 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.930865    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ba67f78-6c99-4e78-ba87-529e861d135d" containerName="util"
Sep 16 18:49:42.930898 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.930877    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ba67f78-6c99-4e78-ba87-529e861d135d" containerName="extract"
Sep 16 18:49:42.930898 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.930885    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ba67f78-6c99-4e78-ba87-529e861d135d" containerName="extract"
Sep 16 18:49:42.930898 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.930896    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ba67f78-6c99-4e78-ba87-529e861d135d" containerName="pull"
Sep 16 18:49:42.931178 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.930905    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ba67f78-6c99-4e78-ba87-529e861d135d" containerName="pull"
Sep 16 18:49:42.931178 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.930999    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ba67f78-6c99-4e78-ba87-529e861d135d" containerName="extract"
Sep 16 18:49:42.933275 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.933252    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss"
Sep 16 18:49:42.937310 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.937287    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-jgf2t\""
Sep 16 18:49:42.937713 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.937696    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:49:42.937787 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.937758    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:49:42.937919 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.937901    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:49:42.949884 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.949858    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss"]
Sep 16 18:49:42.971188 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.971157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxqnr\" (UniqueName: \"kubernetes.io/projected/53db0854-aca7-4f8a-b707-aef59ca9cfca-kube-api-access-nxqnr\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-j48ss\" (UID: \"53db0854-aca7-4f8a-b707-aef59ca9cfca\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss"
Sep 16 18:49:42.971335 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:42.971244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/53db0854-aca7-4f8a-b707-aef59ca9cfca-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-j48ss\" (UID: \"53db0854-aca7-4f8a-b707-aef59ca9cfca\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss"
Sep 16 18:49:43.072730 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:43.072691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqnr\" (UniqueName: \"kubernetes.io/projected/53db0854-aca7-4f8a-b707-aef59ca9cfca-kube-api-access-nxqnr\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-j48ss\" (UID: \"53db0854-aca7-4f8a-b707-aef59ca9cfca\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss"
Sep 16 18:49:43.072916 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:43.072755    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/53db0854-aca7-4f8a-b707-aef59ca9cfca-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-j48ss\" (UID: \"53db0854-aca7-4f8a-b707-aef59ca9cfca\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss"
Sep 16 18:49:43.080362 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:43.080325    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/53db0854-aca7-4f8a-b707-aef59ca9cfca-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-j48ss\" (UID: \"53db0854-aca7-4f8a-b707-aef59ca9cfca\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss"
Sep 16 18:49:43.092413 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:43.092389    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxqnr\" (UniqueName: \"kubernetes.io/projected/53db0854-aca7-4f8a-b707-aef59ca9cfca-kube-api-access-nxqnr\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-j48ss\" (UID: \"53db0854-aca7-4f8a-b707-aef59ca9cfca\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss"
Sep 16 18:49:43.242903 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:43.242860    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss"
Sep 16 18:49:43.534097 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:43.534074    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss"]
Sep 16 18:49:43.535875 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:49:43.535847    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53db0854_aca7_4f8a_b707_aef59ca9cfca.slice/crio-405914427b686d04d9cda6ce118e83143abff8286cbb40c75687b15cea10b146 WatchSource:0}: Error finding container 405914427b686d04d9cda6ce118e83143abff8286cbb40c75687b15cea10b146: Status 404 returned error can't find the container with id 405914427b686d04d9cda6ce118e83143abff8286cbb40c75687b15cea10b146
Sep 16 18:49:43.973617 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:43.973579    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss" event={"ID":"53db0854-aca7-4f8a-b707-aef59ca9cfca","Type":"ContainerStarted","Data":"405914427b686d04d9cda6ce118e83143abff8286cbb40c75687b15cea10b146"}
Sep 16 18:49:46.990391 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:46.990351    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss" event={"ID":"53db0854-aca7-4f8a-b707-aef59ca9cfca","Type":"ContainerStarted","Data":"a1dc33fb98f32f3d06aead40fa554a30b8203c3b2ee285f8c4c9220742c44f88"}
Sep 16 18:49:46.990801 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:46.990485    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss"
Sep 16 18:49:47.413610 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.413576    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:47.652241 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.652206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:47.657963 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.657912    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss" podStartSLOduration=2.348061983 podStartE2EDuration="5.657895898s" totalImagesPullingTime="3.309833915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:47.022818033 +0000 UTC m=+286.825536249" watchObservedRunningTime="2026-09-16 18:49:47.657895898 +0000 UTC m=+287.460614114"
Sep 16 18:49:47.658812 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.658793    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-4d229"]
Sep 16 18:49:47.662266 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.662247    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:49:47.665099 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.665020    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 16 18:49:47.665209 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.665091    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:49:47.665209 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.665111    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-jxz69\""
Sep 16 18:49:47.672833 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.672802    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-4d229"]
Sep 16 18:49:47.716389 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.716352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbrcp\" (UniqueName: \"kubernetes.io/projected/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-kube-api-access-xbrcp\") pod \"keda-operator-579c8958f7-4d229\" (UID: \"1f38d72f-d693-49f6-8d47-e1b6dfb82c7d\") " pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:49:47.716389 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.716395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-cabundle0\") pod \"keda-operator-579c8958f7-4d229\" (UID: \"1f38d72f-d693-49f6-8d47-e1b6dfb82c7d\") " pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:49:47.716624 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.716498    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-certificates\") pod \"keda-operator-579c8958f7-4d229\" (UID: \"1f38d72f-d693-49f6-8d47-e1b6dfb82c7d\") " pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:49:47.817769 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.817737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-cabundle0\") pod \"keda-operator-579c8958f7-4d229\" (UID: \"1f38d72f-d693-49f6-8d47-e1b6dfb82c7d\") " pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:49:47.817949 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.817796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-certificates\") pod \"keda-operator-579c8958f7-4d229\" (UID: \"1f38d72f-d693-49f6-8d47-e1b6dfb82c7d\") " pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:49:47.817949 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.817836    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbrcp\" (UniqueName: \"kubernetes.io/projected/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-kube-api-access-xbrcp\") pod \"keda-operator-579c8958f7-4d229\" (UID: \"1f38d72f-d693-49f6-8d47-e1b6dfb82c7d\") " pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:49:47.818026 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:49:47.817986    2561 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:49:47.818026 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:49:47.818011    2561 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:49:47.818026 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:49:47.818024    2561 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-4d229: references non-existent secret key: ca.crt
Sep 16 18:49:47.818151 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:49:47.818129    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-certificates podName:1f38d72f-d693-49f6-8d47-e1b6dfb82c7d nodeName:}" failed. No retries permitted until 2026-09-16 18:49:48.318106511 +0000 UTC m=+288.120824710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-certificates") pod "keda-operator-579c8958f7-4d229" (UID: "1f38d72f-d693-49f6-8d47-e1b6dfb82c7d") : references non-existent secret key: ca.crt
Sep 16 18:49:47.818437 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.818419    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-cabundle0\") pod \"keda-operator-579c8958f7-4d229\" (UID: \"1f38d72f-d693-49f6-8d47-e1b6dfb82c7d\") " pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:49:47.838839 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.838801    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbrcp\" (UniqueName: \"kubernetes.io/projected/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-kube-api-access-xbrcp\") pod \"keda-operator-579c8958f7-4d229\" (UID: \"1f38d72f-d693-49f6-8d47-e1b6dfb82c7d\") " pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:49:47.845105 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:47.845079    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:48.265890 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.265858    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:48.321002 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.320960    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-certificates\") pod \"keda-operator-579c8958f7-4d229\" (UID: \"1f38d72f-d693-49f6-8d47-e1b6dfb82c7d\") " pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:49:48.321181 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:49:48.321145    2561 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:49:48.321181 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:49:48.321162    2561 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:49:48.321181 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:49:48.321172    2561 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-4d229: references non-existent secret key: ca.crt
Sep 16 18:49:48.321286 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:49:48.321239    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-certificates podName:1f38d72f-d693-49f6-8d47-e1b6dfb82c7d nodeName:}" failed. No retries permitted until 2026-09-16 18:49:49.321220852 +0000 UTC m=+289.123939063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-certificates") pod "keda-operator-579c8958f7-4d229" (UID: "1f38d72f-d693-49f6-8d47-e1b6dfb82c7d") : references non-existent secret key: ca.crt
Sep 16 18:49:48.551148 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.551066    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"]
Sep 16 18:49:48.554379 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.554363    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"
Sep 16 18:49:48.557256 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.557232    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 16 18:49:48.566009 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.565984    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"]
Sep 16 18:49:48.602969 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.602938    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:48.623674 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.623634    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c21fae12-313c-4f68-8cb4-90babb720cc3-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-cljmd\" (UID: \"c21fae12-313c-4f68-8cb4-90babb720cc3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"
Sep 16 18:49:48.623805 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.623680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c21fae12-313c-4f68-8cb4-90babb720cc3-certificates\") pod \"keda-metrics-apiserver-77857b7b98-cljmd\" (UID: \"c21fae12-313c-4f68-8cb4-90babb720cc3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"
Sep 16 18:49:48.623805 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.623739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djglg\" (UniqueName: \"kubernetes.io/projected/c21fae12-313c-4f68-8cb4-90babb720cc3-kube-api-access-djglg\") pod \"keda-metrics-apiserver-77857b7b98-cljmd\" (UID: \"c21fae12-313c-4f68-8cb4-90babb720cc3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"
Sep 16 18:49:48.724950 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.724914    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c21fae12-313c-4f68-8cb4-90babb720cc3-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-cljmd\" (UID: \"c21fae12-313c-4f68-8cb4-90babb720cc3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"
Sep 16 18:49:48.724950 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.724949    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c21fae12-313c-4f68-8cb4-90babb720cc3-certificates\") pod \"keda-metrics-apiserver-77857b7b98-cljmd\" (UID: \"c21fae12-313c-4f68-8cb4-90babb720cc3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"
Sep 16 18:49:48.725181 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.725112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djglg\" (UniqueName: \"kubernetes.io/projected/c21fae12-313c-4f68-8cb4-90babb720cc3-kube-api-access-djglg\") pod \"keda-metrics-apiserver-77857b7b98-cljmd\" (UID: \"c21fae12-313c-4f68-8cb4-90babb720cc3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"
Sep 16 18:49:48.725395 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.725377    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c21fae12-313c-4f68-8cb4-90babb720cc3-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-cljmd\" (UID: \"c21fae12-313c-4f68-8cb4-90babb720cc3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"
Sep 16 18:49:48.727861 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.727832    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c21fae12-313c-4f68-8cb4-90babb720cc3-certificates\") pod \"keda-metrics-apiserver-77857b7b98-cljmd\" (UID: \"c21fae12-313c-4f68-8cb4-90babb720cc3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"
Sep 16 18:49:48.735019 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.734998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djglg\" (UniqueName: \"kubernetes.io/projected/c21fae12-313c-4f68-8cb4-90babb720cc3-kube-api-access-djglg\") pod \"keda-metrics-apiserver-77857b7b98-cljmd\" (UID: \"c21fae12-313c-4f68-8cb4-90babb720cc3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"
Sep 16 18:49:48.866203 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.866105    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"
Sep 16 18:49:48.950331 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.950294    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:48.960249 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.960216    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-z72lz"]
Sep 16 18:49:48.963662 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.963636    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-z72lz"
Sep 16 18:49:48.967644 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.967622    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 18:49:48.990464 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:48.990430    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-z72lz"]
Sep 16 18:49:49.027146 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.027114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/97dc0e90-8a7e-4023-88fb-9909741f6f4b-certificates\") pod \"keda-admission-6c4d864b86-z72lz\" (UID: \"97dc0e90-8a7e-4023-88fb-9909741f6f4b\") " pod="openshift-keda/keda-admission-6c4d864b86-z72lz"
Sep 16 18:49:49.027326 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.027162    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxlqg\" (UniqueName: \"kubernetes.io/projected/97dc0e90-8a7e-4023-88fb-9909741f6f4b-kube-api-access-bxlqg\") pod \"keda-admission-6c4d864b86-z72lz\" (UID: \"97dc0e90-8a7e-4023-88fb-9909741f6f4b\") " pod="openshift-keda/keda-admission-6c4d864b86-z72lz"
Sep 16 18:49:49.113262 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.113218    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:49.128107 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.128007    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/97dc0e90-8a7e-4023-88fb-9909741f6f4b-certificates\") pod \"keda-admission-6c4d864b86-z72lz\" (UID: \"97dc0e90-8a7e-4023-88fb-9909741f6f4b\") " pod="openshift-keda/keda-admission-6c4d864b86-z72lz"
Sep 16 18:49:49.128107 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.128090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxlqg\" (UniqueName: \"kubernetes.io/projected/97dc0e90-8a7e-4023-88fb-9909741f6f4b-kube-api-access-bxlqg\") pod \"keda-admission-6c4d864b86-z72lz\" (UID: \"97dc0e90-8a7e-4023-88fb-9909741f6f4b\") " pod="openshift-keda/keda-admission-6c4d864b86-z72lz"
Sep 16 18:49:49.128273 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:49:49.128176    2561 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 16 18:49:49.128273 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:49:49.128202    2561 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-z72lz: secret "keda-admission-webhooks-certs" not found
Sep 16 18:49:49.128273 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:49:49.128255    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97dc0e90-8a7e-4023-88fb-9909741f6f4b-certificates podName:97dc0e90-8a7e-4023-88fb-9909741f6f4b nodeName:}" failed. No retries permitted until 2026-09-16 18:49:49.628239659 +0000 UTC m=+289.430957854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/97dc0e90-8a7e-4023-88fb-9909741f6f4b-certificates") pod "keda-admission-6c4d864b86-z72lz" (UID: "97dc0e90-8a7e-4023-88fb-9909741f6f4b") : secret "keda-admission-webhooks-certs" not found
Sep 16 18:49:49.140627 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.140591    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxlqg\" (UniqueName: \"kubernetes.io/projected/97dc0e90-8a7e-4023-88fb-9909741f6f4b-kube-api-access-bxlqg\") pod \"keda-admission-6c4d864b86-z72lz\" (UID: \"97dc0e90-8a7e-4023-88fb-9909741f6f4b\") " pod="openshift-keda/keda-admission-6c4d864b86-z72lz"
Sep 16 18:49:49.145671 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.145649    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"]
Sep 16 18:49:49.148244 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:49:49.148210    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21fae12_313c_4f68_8cb4_90babb720cc3.slice/crio-fe2b295037ec05d60e7e575e2e6a200338e29fb130fa7df74d4f260d2ab6317b WatchSource:0}: Error finding container fe2b295037ec05d60e7e575e2e6a200338e29fb130fa7df74d4f260d2ab6317b: Status 404 returned error can't find the container with id fe2b295037ec05d60e7e575e2e6a200338e29fb130fa7df74d4f260d2ab6317b
Sep 16 18:49:49.276739 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.276688    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:49.330697 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.330653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-certificates\") pod \"keda-operator-579c8958f7-4d229\" (UID: \"1f38d72f-d693-49f6-8d47-e1b6dfb82c7d\") " pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:49:49.333398 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.333371    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1f38d72f-d693-49f6-8d47-e1b6dfb82c7d-certificates\") pod \"keda-operator-579c8958f7-4d229\" (UID: \"1f38d72f-d693-49f6-8d47-e1b6dfb82c7d\") " pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:49:49.473506 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.473471    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:49:49.485486 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.485461    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:49.633638 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.633602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/97dc0e90-8a7e-4023-88fb-9909741f6f4b-certificates\") pod \"keda-admission-6c4d864b86-z72lz\" (UID: \"97dc0e90-8a7e-4023-88fb-9909741f6f4b\") " pod="openshift-keda/keda-admission-6c4d864b86-z72lz"
Sep 16 18:49:49.636275 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.636245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/97dc0e90-8a7e-4023-88fb-9909741f6f4b-certificates\") pod \"keda-admission-6c4d864b86-z72lz\" (UID: \"97dc0e90-8a7e-4023-88fb-9909741f6f4b\") " pod="openshift-keda/keda-admission-6c4d864b86-z72lz"
Sep 16 18:49:49.741827 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.741747    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:49.752572 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.752543    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-4d229"]
Sep 16 18:49:49.755489 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:49:49.755456    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f38d72f_d693_49f6_8d47_e1b6dfb82c7d.slice/crio-e43ddebc18f73d662ad7a3bafab2c0bb3251ccc574e9b5631775a723b4a28189 WatchSource:0}: Error finding container e43ddebc18f73d662ad7a3bafab2c0bb3251ccc574e9b5631775a723b4a28189: Status 404 returned error can't find the container with id e43ddebc18f73d662ad7a3bafab2c0bb3251ccc574e9b5631775a723b4a28189
Sep 16 18:49:49.874169 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.874131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-z72lz"
Sep 16 18:49:49.938121 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:49.938092    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:50.005091 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:50.004989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-4d229" event={"ID":"1f38d72f-d693-49f6-8d47-e1b6dfb82c7d","Type":"ContainerStarted","Data":"e43ddebc18f73d662ad7a3bafab2c0bb3251ccc574e9b5631775a723b4a28189"}
Sep 16 18:49:50.006209 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:50.006185    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd" event={"ID":"c21fae12-313c-4f68-8cb4-90babb720cc3","Type":"ContainerStarted","Data":"fe2b295037ec05d60e7e575e2e6a200338e29fb130fa7df74d4f260d2ab6317b"}
Sep 16 18:49:50.040964 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:50.040899    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:50.187899 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:50.187867    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-z72lz"]
Sep 16 18:49:50.191925 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:49:50.191886    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97dc0e90_8a7e_4023_88fb_9909741f6f4b.slice/crio-06390f8dfb7e9b9e685eabb10c0c69dc7c77589b8ab68e2e7018a0248dc8810a WatchSource:0}: Error finding container 06390f8dfb7e9b9e685eabb10c0c69dc7c77589b8ab68e2e7018a0248dc8810a: Status 404 returned error can't find the container with id 06390f8dfb7e9b9e685eabb10c0c69dc7c77589b8ab68e2e7018a0248dc8810a
Sep 16 18:49:51.021522 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:51.021471    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-z72lz" event={"ID":"97dc0e90-8a7e-4023-88fb-9909741f6f4b","Type":"ContainerStarted","Data":"06390f8dfb7e9b9e685eabb10c0c69dc7c77589b8ab68e2e7018a0248dc8810a"}
Sep 16 18:49:52.028461 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:52.028360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-z72lz" event={"ID":"97dc0e90-8a7e-4023-88fb-9909741f6f4b","Type":"ContainerStarted","Data":"a73ee190e66dc5670699d4a75e35bfaaf2a2a8c3275bf639c74672f9cb20b3d6"}
Sep 16 18:49:52.029081 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:52.028454    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-z72lz"
Sep 16 18:49:52.047355 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:52.047188    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-z72lz" podStartSLOduration=2.5158118800000002 podStartE2EDuration="4.04716805s" totalImagesPullingTime="1.53135617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:52.046538165 +0000 UTC m=+291.849256384" watchObservedRunningTime="2026-09-16 18:49:52.04716805 +0000 UTC m=+291.849886269"
Sep 16 18:49:54.040362 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:54.040326    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-4d229" event={"ID":"1f38d72f-d693-49f6-8d47-e1b6dfb82c7d","Type":"ContainerStarted","Data":"4c81224966b0a6c1247a7b3fcfbe03c91755fd63fa7d433b610eab06b7225e37"}
Sep 16 18:49:54.040765 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:54.040468    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:49:54.058777 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:49:54.058729    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-4d229" podStartSLOduration=3.452054945 podStartE2EDuration="7.058710626s" totalImagesPullingTime="3.606655681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:54.058209158 +0000 UTC m=+293.860927387" watchObservedRunningTime="2026-09-16 18:49:54.058710626 +0000 UTC m=+293.861428865"
Sep 16 18:50:00.613988 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:00.613960    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 18:50:00.619006 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:00.618978    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 18:50:00.621397 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:00.621368    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:50:07.102247 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:07.102207    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd" event={"ID":"c21fae12-313c-4f68-8cb4-90babb720cc3","Type":"ContainerStarted","Data":"779308f0109a0e6aef66a2bb7660ffab4db7c584c67a8f65fcff42f7ede19c97"}
Sep 16 18:50:07.102709 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:07.102369    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"
Sep 16 18:50:07.126719 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:07.126657    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd" podStartSLOduration=1.727928963 podStartE2EDuration="19.126639796s" totalImagesPullingTime="17.398710833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:07.124157337 +0000 UTC m=+306.926875552" watchObservedRunningTime="2026-09-16 18:50:07.126639796 +0000 UTC m=+306.929358012"
Sep 16 18:50:07.997934 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:07.997900    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-j48ss"
Sep 16 18:50:13.037105 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:13.037073    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-z72lz"
Sep 16 18:50:15.047341 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:15.047312    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-4d229"
Sep 16 18:50:18.113075 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:18.113035    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-cljmd"
Sep 16 18:50:26.932199 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:26.932161    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:27.030359 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:27.030326    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:38.276563 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.276525    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:38.539071 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.538974    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"]
Sep 16 18:50:38.543443 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.543423    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"
Sep 16 18:50:38.546489 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.546468    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:50:38.546682 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.546669    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:50:38.547767 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.547746    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v2xds\""
Sep 16 18:50:38.556556 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.556527    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"]
Sep 16 18:50:38.657587 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.657546    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwz4q\" (UniqueName: \"kubernetes.io/projected/a4a39880-4eea-4535-86c0-73a109676567-kube-api-access-lwz4q\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw\" (UID: \"a4a39880-4eea-4535-86c0-73a109676567\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"
Sep 16 18:50:38.657778 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.657613    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4a39880-4eea-4535-86c0-73a109676567-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw\" (UID: \"a4a39880-4eea-4535-86c0-73a109676567\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"
Sep 16 18:50:38.657778 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.657678    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4a39880-4eea-4535-86c0-73a109676567-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw\" (UID: \"a4a39880-4eea-4535-86c0-73a109676567\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"
Sep 16 18:50:38.758316 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.758285    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwz4q\" (UniqueName: \"kubernetes.io/projected/a4a39880-4eea-4535-86c0-73a109676567-kube-api-access-lwz4q\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw\" (UID: \"a4a39880-4eea-4535-86c0-73a109676567\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"
Sep 16 18:50:38.758498 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.758325    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4a39880-4eea-4535-86c0-73a109676567-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw\" (UID: \"a4a39880-4eea-4535-86c0-73a109676567\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"
Sep 16 18:50:38.758498 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.758369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4a39880-4eea-4535-86c0-73a109676567-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw\" (UID: \"a4a39880-4eea-4535-86c0-73a109676567\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"
Sep 16 18:50:38.758666 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.758650    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4a39880-4eea-4535-86c0-73a109676567-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw\" (UID: \"a4a39880-4eea-4535-86c0-73a109676567\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"
Sep 16 18:50:38.758752 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.758732    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4a39880-4eea-4535-86c0-73a109676567-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw\" (UID: \"a4a39880-4eea-4535-86c0-73a109676567\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"
Sep 16 18:50:38.768800 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.768765    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwz4q\" (UniqueName: \"kubernetes.io/projected/a4a39880-4eea-4535-86c0-73a109676567-kube-api-access-lwz4q\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw\" (UID: \"a4a39880-4eea-4535-86c0-73a109676567\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"
Sep 16 18:50:38.852606 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:38.852514    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"
Sep 16 18:50:39.144034 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:39.143996    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"]
Sep 16 18:50:39.145496 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:50:39.145464    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a39880_4eea_4535_86c0_73a109676567.slice/crio-56aa1f565fffbd6175a1e86a257d2664eabaade039bd60aacaaeab9429c880d5 WatchSource:0}: Error finding container 56aa1f565fffbd6175a1e86a257d2664eabaade039bd60aacaaeab9429c880d5: Status 404 returned error can't find the container with id 56aa1f565fffbd6175a1e86a257d2664eabaade039bd60aacaaeab9429c880d5
Sep 16 18:50:39.259440 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:39.259407    2561 generic.go:356] "Generic (PLEG): container finished" podID="a4a39880-4eea-4535-86c0-73a109676567" containerID="2c9766dff0d1ae0a6f7cff01a290083c628ea13fe38c20d5be0dbe565167b62a" exitCode=0
Sep 16 18:50:39.259592 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:39.259485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw" event={"ID":"a4a39880-4eea-4535-86c0-73a109676567","Type":"ContainerDied","Data":"2c9766dff0d1ae0a6f7cff01a290083c628ea13fe38c20d5be0dbe565167b62a"}
Sep 16 18:50:39.259592 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:39.259532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw" event={"ID":"a4a39880-4eea-4535-86c0-73a109676567","Type":"ContainerStarted","Data":"56aa1f565fffbd6175a1e86a257d2664eabaade039bd60aacaaeab9429c880d5"}
Sep 16 18:50:39.260531 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:39.260511    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:50:39.689555 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:39.689520    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:41.271698 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:41.271645    2561 generic.go:356] "Generic (PLEG): container finished" podID="a4a39880-4eea-4535-86c0-73a109676567" containerID="49f3eb5cef0a7a23445374c681af89217d3b0651d5fc0dbd4642bfead93721cb" exitCode=0
Sep 16 18:50:41.272121 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:41.271704    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw" event={"ID":"a4a39880-4eea-4535-86c0-73a109676567","Type":"ContainerDied","Data":"49f3eb5cef0a7a23445374c681af89217d3b0651d5fc0dbd4642bfead93721cb"}
Sep 16 18:50:42.280044 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:42.280002    2561 generic.go:356] "Generic (PLEG): container finished" podID="a4a39880-4eea-4535-86c0-73a109676567" containerID="05de03d504654bcc0d084490cc683d276ece8fd249f3b77767302370685c6297" exitCode=0
Sep 16 18:50:42.280439 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:42.280089    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw" event={"ID":"a4a39880-4eea-4535-86c0-73a109676567","Type":"ContainerDied","Data":"05de03d504654bcc0d084490cc683d276ece8fd249f3b77767302370685c6297"}
Sep 16 18:50:43.410966 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:43.410941    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"
Sep 16 18:50:43.499129 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:43.499073    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwz4q\" (UniqueName: \"kubernetes.io/projected/a4a39880-4eea-4535-86c0-73a109676567-kube-api-access-lwz4q\") pod \"a4a39880-4eea-4535-86c0-73a109676567\" (UID: \"a4a39880-4eea-4535-86c0-73a109676567\") "
Sep 16 18:50:43.499318 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:43.499189    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4a39880-4eea-4535-86c0-73a109676567-util\") pod \"a4a39880-4eea-4535-86c0-73a109676567\" (UID: \"a4a39880-4eea-4535-86c0-73a109676567\") "
Sep 16 18:50:43.499318 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:43.499216    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4a39880-4eea-4535-86c0-73a109676567-bundle\") pod \"a4a39880-4eea-4535-86c0-73a109676567\" (UID: \"a4a39880-4eea-4535-86c0-73a109676567\") "
Sep 16 18:50:43.499916 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:43.499888    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a39880-4eea-4535-86c0-73a109676567-bundle" (OuterVolumeSpecName: "bundle") pod "a4a39880-4eea-4535-86c0-73a109676567" (UID: "a4a39880-4eea-4535-86c0-73a109676567"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:43.501408 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:43.501383    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a39880-4eea-4535-86c0-73a109676567-kube-api-access-lwz4q" (OuterVolumeSpecName: "kube-api-access-lwz4q") pod "a4a39880-4eea-4535-86c0-73a109676567" (UID: "a4a39880-4eea-4535-86c0-73a109676567"). InnerVolumeSpecName "kube-api-access-lwz4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:43.504746 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:43.504721    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a39880-4eea-4535-86c0-73a109676567-util" (OuterVolumeSpecName: "util") pod "a4a39880-4eea-4535-86c0-73a109676567" (UID: "a4a39880-4eea-4535-86c0-73a109676567"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:43.600302 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:43.600206    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4a39880-4eea-4535-86c0-73a109676567-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:50:43.600302 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:43.600250    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwz4q\" (UniqueName: \"kubernetes.io/projected/a4a39880-4eea-4535-86c0-73a109676567-kube-api-access-lwz4q\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:50:43.600302 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:43.600260    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4a39880-4eea-4535-86c0-73a109676567-util\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:50:44.291221 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:44.291190    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw"
Sep 16 18:50:44.291371 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:44.291187    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405clfw" event={"ID":"a4a39880-4eea-4535-86c0-73a109676567","Type":"ContainerDied","Data":"56aa1f565fffbd6175a1e86a257d2664eabaade039bd60aacaaeab9429c880d5"}
Sep 16 18:50:44.291371 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:44.291312    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56aa1f565fffbd6175a1e86a257d2664eabaade039bd60aacaaeab9429c880d5"
Sep 16 18:50:46.055201 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:46.055163    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:46.120087 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:46.120029    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:49.664799 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:49.664761    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:50.955805 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:50.955771    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:51.740546 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.737658    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5qb8x"]
Sep 16 18:50:51.740546 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.738487    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4a39880-4eea-4535-86c0-73a109676567" containerName="pull"
Sep 16 18:50:51.740546 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.738509    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4a39880-4eea-4535-86c0-73a109676567" containerName="pull"
Sep 16 18:50:51.740546 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.738535    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4a39880-4eea-4535-86c0-73a109676567" containerName="util"
Sep 16 18:50:51.740546 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.738543    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4a39880-4eea-4535-86c0-73a109676567" containerName="util"
Sep 16 18:50:51.740546 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.738589    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4a39880-4eea-4535-86c0-73a109676567" containerName="extract"
Sep 16 18:50:51.740546 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.738597    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4a39880-4eea-4535-86c0-73a109676567" containerName="extract"
Sep 16 18:50:51.740546 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.738754    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4a39880-4eea-4535-86c0-73a109676567" containerName="extract"
Sep 16 18:50:51.742846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.742307    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5qb8x"
Sep 16 18:50:51.746250 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.746230    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xwlrf\""
Sep 16 18:50:51.746428 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.746414    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 18:50:51.746577 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.746565    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:50:51.764605 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.764577    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5qb8x"]
Sep 16 18:50:51.871712 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.871681    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq5v6\" (UniqueName: \"kubernetes.io/projected/b326f906-08b3-4bcc-a305-b7653daab959-kube-api-access-gq5v6\") pod \"cert-manager-operator-controller-manager-684f79c9b9-5qb8x\" (UID: \"b326f906-08b3-4bcc-a305-b7653daab959\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5qb8x"
Sep 16 18:50:51.871891 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.871732    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b326f906-08b3-4bcc-a305-b7653daab959-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-5qb8x\" (UID: \"b326f906-08b3-4bcc-a305-b7653daab959\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5qb8x"
Sep 16 18:50:51.973271 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.973235    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq5v6\" (UniqueName: \"kubernetes.io/projected/b326f906-08b3-4bcc-a305-b7653daab959-kube-api-access-gq5v6\") pod \"cert-manager-operator-controller-manager-684f79c9b9-5qb8x\" (UID: \"b326f906-08b3-4bcc-a305-b7653daab959\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5qb8x"
Sep 16 18:50:51.973665 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.973289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b326f906-08b3-4bcc-a305-b7653daab959-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-5qb8x\" (UID: \"b326f906-08b3-4bcc-a305-b7653daab959\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5qb8x"
Sep 16 18:50:51.973665 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.973644    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b326f906-08b3-4bcc-a305-b7653daab959-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-5qb8x\" (UID: \"b326f906-08b3-4bcc-a305-b7653daab959\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5qb8x"
Sep 16 18:50:51.982762 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:51.982731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq5v6\" (UniqueName: \"kubernetes.io/projected/b326f906-08b3-4bcc-a305-b7653daab959-kube-api-access-gq5v6\") pod \"cert-manager-operator-controller-manager-684f79c9b9-5qb8x\" (UID: \"b326f906-08b3-4bcc-a305-b7653daab959\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5qb8x"
Sep 16 18:50:52.051825 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:52.051705    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5qb8x"
Sep 16 18:50:52.344103 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:52.344075    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5qb8x"]
Sep 16 18:50:52.346593 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:50:52.346562    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb326f906_08b3_4bcc_a305_b7653daab959.slice/crio-ad9ae8cfb7ef742ba02a434a1f15747593cd885a0c5d59bf714984fddd4f2a81 WatchSource:0}: Error finding container ad9ae8cfb7ef742ba02a434a1f15747593cd885a0c5d59bf714984fddd4f2a81: Status 404 returned error can't find the container with id ad9ae8cfb7ef742ba02a434a1f15747593cd885a0c5d59bf714984fddd4f2a81
Sep 16 18:50:53.334748 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:53.334716    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5qb8x" event={"ID":"b326f906-08b3-4bcc-a305-b7653daab959","Type":"ContainerStarted","Data":"ad9ae8cfb7ef742ba02a434a1f15747593cd885a0c5d59bf714984fddd4f2a81"}
Sep 16 18:50:53.467249 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:53.467203    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:53.564985 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:53.564940    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:54.341608 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:54.341571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5qb8x" event={"ID":"b326f906-08b3-4bcc-a305-b7653daab959","Type":"ContainerStarted","Data":"97d8fb3ba83b750e05be4df711f7e6b9108da8ef47fa43c7c2154a8424556385"}
Sep 16 18:50:54.368410 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:54.368349    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5qb8x" podStartSLOduration=1.895045224 podStartE2EDuration="3.368326537s" totalImagesPullingTime="1.473281313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:54.365469102 +0000 UTC m=+354.168187355" watchObservedRunningTime="2026-09-16 18:50:54.368326537 +0000 UTC m=+354.171044753"
Sep 16 18:50:55.504935 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:55.504901    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:55.610121 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:55.610079    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:57.675913 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:57.675871    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:58.100611 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:58.100571    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:58.490966 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:58.490926    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:58.568238 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:58.568206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:59.681876 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:59.681827    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:59.816284 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:59.816245    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:59.925349 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:50:59.925287    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:00.778793 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.778766    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"]
Sep 16 18:51:00.782468 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.782450    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"
Sep 16 18:51:00.788462 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.788439    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v2xds\""
Sep 16 18:51:00.788462 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.788456    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:51:00.789590 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.789570    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:51:00.804345 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.804322    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"]
Sep 16 18:51:00.850642 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.850605    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c36c3aa-db6a-42a5-bca5-d005452e951b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl\" (UID: \"6c36c3aa-db6a-42a5-bca5-d005452e951b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"
Sep 16 18:51:00.850846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.850652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfmxw\" (UniqueName: \"kubernetes.io/projected/6c36c3aa-db6a-42a5-bca5-d005452e951b-kube-api-access-gfmxw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl\" (UID: \"6c36c3aa-db6a-42a5-bca5-d005452e951b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"
Sep 16 18:51:00.850846 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.850768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c36c3aa-db6a-42a5-bca5-d005452e951b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl\" (UID: \"6c36c3aa-db6a-42a5-bca5-d005452e951b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"
Sep 16 18:51:00.951996 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.951959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfmxw\" (UniqueName: \"kubernetes.io/projected/6c36c3aa-db6a-42a5-bca5-d005452e951b-kube-api-access-gfmxw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl\" (UID: \"6c36c3aa-db6a-42a5-bca5-d005452e951b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"
Sep 16 18:51:00.952197 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.952022    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c36c3aa-db6a-42a5-bca5-d005452e951b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl\" (UID: \"6c36c3aa-db6a-42a5-bca5-d005452e951b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"
Sep 16 18:51:00.952197 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.952122    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c36c3aa-db6a-42a5-bca5-d005452e951b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl\" (UID: \"6c36c3aa-db6a-42a5-bca5-d005452e951b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"
Sep 16 18:51:00.952519 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.952495    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c36c3aa-db6a-42a5-bca5-d005452e951b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl\" (UID: \"6c36c3aa-db6a-42a5-bca5-d005452e951b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"
Sep 16 18:51:00.952566 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.952505    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c36c3aa-db6a-42a5-bca5-d005452e951b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl\" (UID: \"6c36c3aa-db6a-42a5-bca5-d005452e951b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"
Sep 16 18:51:00.961169 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:00.961143    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfmxw\" (UniqueName: \"kubernetes.io/projected/6c36c3aa-db6a-42a5-bca5-d005452e951b-kube-api-access-gfmxw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl\" (UID: \"6c36c3aa-db6a-42a5-bca5-d005452e951b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"
Sep 16 18:51:01.091142 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:01.091037    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"
Sep 16 18:51:01.372277 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:01.372193    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"]
Sep 16 18:51:01.377629 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:51:01.377599    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c36c3aa_db6a_42a5_bca5_d005452e951b.slice/crio-d05c389cde69a19241ee15ae41160a3918382a7f9e62f39edcc03d9f7117a8cd WatchSource:0}: Error finding container d05c389cde69a19241ee15ae41160a3918382a7f9e62f39edcc03d9f7117a8cd: Status 404 returned error can't find the container with id d05c389cde69a19241ee15ae41160a3918382a7f9e62f39edcc03d9f7117a8cd
Sep 16 18:51:02.389378 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:02.389282    2561 generic.go:356] "Generic (PLEG): container finished" podID="6c36c3aa-db6a-42a5-bca5-d005452e951b" containerID="d9ea2f73f9c051a13ca802c1e3c04f5584aac30200899407f3beecb9823b9799" exitCode=0
Sep 16 18:51:02.389785 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:02.389372    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl" event={"ID":"6c36c3aa-db6a-42a5-bca5-d005452e951b","Type":"ContainerDied","Data":"d9ea2f73f9c051a13ca802c1e3c04f5584aac30200899407f3beecb9823b9799"}
Sep 16 18:51:02.389785 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:02.389462    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl" event={"ID":"6c36c3aa-db6a-42a5-bca5-d005452e951b","Type":"ContainerStarted","Data":"d05c389cde69a19241ee15ae41160a3918382a7f9e62f39edcc03d9f7117a8cd"}
Sep 16 18:51:05.408675 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:05.408587    2561 generic.go:356] "Generic (PLEG): container finished" podID="6c36c3aa-db6a-42a5-bca5-d005452e951b" containerID="2977fafb119e8a572fda36c288d7efe33671dc146d2376d6aa7db2bac7224212" exitCode=0
Sep 16 18:51:05.409037 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:05.408671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl" event={"ID":"6c36c3aa-db6a-42a5-bca5-d005452e951b","Type":"ContainerDied","Data":"2977fafb119e8a572fda36c288d7efe33671dc146d2376d6aa7db2bac7224212"}
Sep 16 18:51:05.809236 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:05.809204    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:06.416383 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:06.416343    2561 generic.go:356] "Generic (PLEG): container finished" podID="6c36c3aa-db6a-42a5-bca5-d005452e951b" containerID="958d008e00f64fc2d0d9e89b8a1a31ed0b6d57fe66b6a83ad25c343741d57c97" exitCode=0
Sep 16 18:51:06.416744 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:06.416428    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl" event={"ID":"6c36c3aa-db6a-42a5-bca5-d005452e951b","Type":"ContainerDied","Data":"958d008e00f64fc2d0d9e89b8a1a31ed0b6d57fe66b6a83ad25c343741d57c97"}
Sep 16 18:51:07.548264 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:07.548240    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"
Sep 16 18:51:07.715327 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:07.715298    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c36c3aa-db6a-42a5-bca5-d005452e951b-bundle\") pod \"6c36c3aa-db6a-42a5-bca5-d005452e951b\" (UID: \"6c36c3aa-db6a-42a5-bca5-d005452e951b\") "
Sep 16 18:51:07.715505 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:07.715343    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c36c3aa-db6a-42a5-bca5-d005452e951b-util\") pod \"6c36c3aa-db6a-42a5-bca5-d005452e951b\" (UID: \"6c36c3aa-db6a-42a5-bca5-d005452e951b\") "
Sep 16 18:51:07.715505 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:07.715384    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfmxw\" (UniqueName: \"kubernetes.io/projected/6c36c3aa-db6a-42a5-bca5-d005452e951b-kube-api-access-gfmxw\") pod \"6c36c3aa-db6a-42a5-bca5-d005452e951b\" (UID: \"6c36c3aa-db6a-42a5-bca5-d005452e951b\") "
Sep 16 18:51:07.715791 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:07.715764    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c36c3aa-db6a-42a5-bca5-d005452e951b-bundle" (OuterVolumeSpecName: "bundle") pod "6c36c3aa-db6a-42a5-bca5-d005452e951b" (UID: "6c36c3aa-db6a-42a5-bca5-d005452e951b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:07.717825 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:07.717796    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c36c3aa-db6a-42a5-bca5-d005452e951b-kube-api-access-gfmxw" (OuterVolumeSpecName: "kube-api-access-gfmxw") pod "6c36c3aa-db6a-42a5-bca5-d005452e951b" (UID: "6c36c3aa-db6a-42a5-bca5-d005452e951b"). InnerVolumeSpecName "kube-api-access-gfmxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:07.720819 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:07.720786    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c36c3aa-db6a-42a5-bca5-d005452e951b-util" (OuterVolumeSpecName: "util") pod "6c36c3aa-db6a-42a5-bca5-d005452e951b" (UID: "6c36c3aa-db6a-42a5-bca5-d005452e951b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:07.816124 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:07.816091    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c36c3aa-db6a-42a5-bca5-d005452e951b-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:07.816124 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:07.816117    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c36c3aa-db6a-42a5-bca5-d005452e951b-util\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:07.816124 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:07.816129    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfmxw\" (UniqueName: \"kubernetes.io/projected/6c36c3aa-db6a-42a5-bca5-d005452e951b-kube-api-access-gfmxw\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:08.428771 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:08.428737    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl"
Sep 16 18:51:08.428957 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:08.428735    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9qwxl" event={"ID":"6c36c3aa-db6a-42a5-bca5-d005452e951b","Type":"ContainerDied","Data":"d05c389cde69a19241ee15ae41160a3918382a7f9e62f39edcc03d9f7117a8cd"}
Sep 16 18:51:08.428957 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:08.428856    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d05c389cde69a19241ee15ae41160a3918382a7f9e62f39edcc03d9f7117a8cd"
Sep 16 18:51:08.611889 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:08.611858    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:08.710612 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:08.710505    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:10.666397 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:10.666350    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:10.767883 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:10.767849    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:21.622863 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:21.622829    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:23.157430 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:23.157396    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:27.799859 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.799829    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:27.818663 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.818625    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"]
Sep 16 18:51:27.819082 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.819065    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c36c3aa-db6a-42a5-bca5-d005452e951b" containerName="pull"
Sep 16 18:51:27.819134 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.819084    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c36c3aa-db6a-42a5-bca5-d005452e951b" containerName="pull"
Sep 16 18:51:27.819134 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.819096    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c36c3aa-db6a-42a5-bca5-d005452e951b" containerName="util"
Sep 16 18:51:27.819134 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.819102    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c36c3aa-db6a-42a5-bca5-d005452e951b" containerName="util"
Sep 16 18:51:27.819134 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.819113    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c36c3aa-db6a-42a5-bca5-d005452e951b" containerName="extract"
Sep 16 18:51:27.819134 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.819119    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c36c3aa-db6a-42a5-bca5-d005452e951b" containerName="extract"
Sep 16 18:51:27.819288 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.819208    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c36c3aa-db6a-42a5-bca5-d005452e951b" containerName="extract"
Sep 16 18:51:27.821340 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.821324    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.825380 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.825345    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 18:51:27.825521 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.825394    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 18:51:27.825521 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.825406    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 18:51:27.825643 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.825422    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:51:27.825711 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.825645    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:51:27.825711 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.825694    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qhwqn\""
Sep 16 18:51:27.825819 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.825735    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:51:27.833200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.833172    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"]
Sep 16 18:51:27.874910 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.874870    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:27.879171 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.879150    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.879271 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.879183    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.879271 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.879204    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.879382 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.879306    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecd256b-c553-4d4a-8ed2-05216f758c01-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.879431 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.879407    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg424\" (UniqueName: \"kubernetes.io/projected/2ecd256b-c553-4d4a-8ed2-05216f758c01-kube-api-access-tg424\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.879472 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.879434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.879472 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.879453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.980447 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.980413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg424\" (UniqueName: \"kubernetes.io/projected/2ecd256b-c553-4d4a-8ed2-05216f758c01-kube-api-access-tg424\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.980447 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.980452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.980700 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.980472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.980700 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.980497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.980700 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.980515    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.980700 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.980683    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.980899 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.980767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecd256b-c553-4d4a-8ed2-05216f758c01-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.981158 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.981135    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.983228 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.983202    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.983228 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.983225    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.983356 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.983235    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecd256b-c553-4d4a-8ed2-05216f758c01-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.983356 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.983240    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.990747 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.990722    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg424\" (UniqueName: \"kubernetes.io/projected/2ecd256b-c553-4d4a-8ed2-05216f758c01-kube-api-access-tg424\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:27.990996 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:27.990976    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2r92v\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:28.130932 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:28.130839    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:28.425470 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:28.425316    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"]
Sep 16 18:51:28.427902 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:51:28.427875    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecd256b_c553_4d4a_8ed2_05216f758c01.slice/crio-950a8b35c980c6b23d351d6a7827eff38ab8b07d6ee76c0432b904e0cece5db3 WatchSource:0}: Error finding container 950a8b35c980c6b23d351d6a7827eff38ab8b07d6ee76c0432b904e0cece5db3: Status 404 returned error can't find the container with id 950a8b35c980c6b23d351d6a7827eff38ab8b07d6ee76c0432b904e0cece5db3
Sep 16 18:51:28.524681 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:28.524638    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v" event={"ID":"2ecd256b-c553-4d4a-8ed2-05216f758c01","Type":"ContainerStarted","Data":"950a8b35c980c6b23d351d6a7827eff38ab8b07d6ee76c0432b904e0cece5db3"}
Sep 16 18:51:29.071708 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:29.071672    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:31.307715 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:31.307675    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 18:51:31.307986 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:31.307749    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 18:51:31.542027 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:31.541991    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v" event={"ID":"2ecd256b-c553-4d4a-8ed2-05216f758c01","Type":"ContainerStarted","Data":"344e3a917d6c89e2624084b4806bc4d150fc1bc4684991f6718753d114ff7800"}
Sep 16 18:51:31.542217 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:31.542120    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:31.566421 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:31.566324    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v" podStartSLOduration=1.689171497 podStartE2EDuration="4.566306689s" totalImagesPullingTime="2.877135192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:31.563269943 +0000 UTC m=+391.365988170" watchObservedRunningTime="2026-09-16 18:51:31.566306689 +0000 UTC m=+391.369024905"
Sep 16 18:51:32.550849 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:32.550816    2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-2r92v container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 18:51:32.551309 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:32.550888    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v" podUID="2ecd256b-c553-4d4a-8ed2-05216f758c01" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:32.604996 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:32.604957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:32.713560 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:32.713521    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:35.549691 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:35.549657    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:51:36.182030 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:36.181996    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:36.268099 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:36.268018    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:37.772316 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:37.772253    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:37.960163 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:37.960128    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:39.263111 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:39.263066    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:43.782674 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:43.782639    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:43.919881 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:43.919846    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:48.383744 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.383705    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"]
Sep 16 18:51:48.390646 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.390621    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"
Sep 16 18:51:48.394496 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.394449    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:51:48.394843 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.394822    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:51:48.395996 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.395974    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v2xds\""
Sep 16 18:51:48.399238 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.399216    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"]
Sep 16 18:51:48.457948 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.457910    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs\" (UID: \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"
Sep 16 18:51:48.458156 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.457958    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4dxb\" (UniqueName: \"kubernetes.io/projected/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-kube-api-access-w4dxb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs\" (UID: \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"
Sep 16 18:51:48.458156 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.458050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs\" (UID: \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"
Sep 16 18:51:48.483853 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.483815    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"]
Sep 16 18:51:48.487604 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.487586    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"
Sep 16 18:51:48.497458 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.497431    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"]
Sep 16 18:51:48.559147 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.559110    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v\" (UID: \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"
Sep 16 18:51:48.559323 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.559158    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs\" (UID: \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"
Sep 16 18:51:48.559323 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.559191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v\" (UID: \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"
Sep 16 18:51:48.559323 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.559223    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4dxb\" (UniqueName: \"kubernetes.io/projected/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-kube-api-access-w4dxb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs\" (UID: \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"
Sep 16 18:51:48.559323 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.559264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccb7m\" (UniqueName: \"kubernetes.io/projected/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-kube-api-access-ccb7m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v\" (UID: \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"
Sep 16 18:51:48.559323 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.559300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs\" (UID: \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"
Sep 16 18:51:48.559506 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.559485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs\" (UID: \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"
Sep 16 18:51:48.559621 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.559603    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs\" (UID: \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"
Sep 16 18:51:48.569336 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.569312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4dxb\" (UniqueName: \"kubernetes.io/projected/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-kube-api-access-w4dxb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs\" (UID: \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"
Sep 16 18:51:48.585453 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.585420    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"]
Sep 16 18:51:48.589203 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.589185    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"
Sep 16 18:51:48.600202 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.600177    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"]
Sep 16 18:51:48.660377 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.660285    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0013c2ef-f446-46a7-b0e7-fc26ca792192-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts\" (UID: \"0013c2ef-f446-46a7-b0e7-fc26ca792192\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"
Sep 16 18:51:48.660377 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.660345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccb7m\" (UniqueName: \"kubernetes.io/projected/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-kube-api-access-ccb7m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v\" (UID: \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"
Sep 16 18:51:48.660572 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.660408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0013c2ef-f446-46a7-b0e7-fc26ca792192-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts\" (UID: \"0013c2ef-f446-46a7-b0e7-fc26ca792192\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"
Sep 16 18:51:48.660572 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.660480    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v\" (UID: \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"
Sep 16 18:51:48.660572 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.660513    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xng8w\" (UniqueName: \"kubernetes.io/projected/0013c2ef-f446-46a7-b0e7-fc26ca792192-kube-api-access-xng8w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts\" (UID: \"0013c2ef-f446-46a7-b0e7-fc26ca792192\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"
Sep 16 18:51:48.660572 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.660546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v\" (UID: \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"
Sep 16 18:51:48.660904 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.660881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v\" (UID: \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"
Sep 16 18:51:48.660943 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.660891    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v\" (UID: \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"
Sep 16 18:51:48.669009 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.668973    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccb7m\" (UniqueName: \"kubernetes.io/projected/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-kube-api-access-ccb7m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v\" (UID: \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"
Sep 16 18:51:48.682383 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.682349    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"]
Sep 16 18:51:48.686218 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.686200    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"
Sep 16 18:51:48.695671 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.695636    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"]
Sep 16 18:51:48.701390 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.701367    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"
Sep 16 18:51:48.761633 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.761600    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng8w\" (UniqueName: \"kubernetes.io/projected/0013c2ef-f446-46a7-b0e7-fc26ca792192-kube-api-access-xng8w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts\" (UID: \"0013c2ef-f446-46a7-b0e7-fc26ca792192\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"
Sep 16 18:51:48.761759 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.761654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs5mp\" (UniqueName: \"kubernetes.io/projected/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-kube-api-access-gs5mp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm\" (UID: \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"
Sep 16 18:51:48.761759 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.761695    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm\" (UID: \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"
Sep 16 18:51:48.761759 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.761719    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm\" (UID: \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"
Sep 16 18:51:48.761926 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.761812    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0013c2ef-f446-46a7-b0e7-fc26ca792192-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts\" (UID: \"0013c2ef-f446-46a7-b0e7-fc26ca792192\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"
Sep 16 18:51:48.761926 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.761884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0013c2ef-f446-46a7-b0e7-fc26ca792192-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts\" (UID: \"0013c2ef-f446-46a7-b0e7-fc26ca792192\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"
Sep 16 18:51:48.762280 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.762257    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0013c2ef-f446-46a7-b0e7-fc26ca792192-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts\" (UID: \"0013c2ef-f446-46a7-b0e7-fc26ca792192\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"
Sep 16 18:51:48.762363 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.762339    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0013c2ef-f446-46a7-b0e7-fc26ca792192-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts\" (UID: \"0013c2ef-f446-46a7-b0e7-fc26ca792192\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"
Sep 16 18:51:48.783039 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.783008    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xng8w\" (UniqueName: \"kubernetes.io/projected/0013c2ef-f446-46a7-b0e7-fc26ca792192-kube-api-access-xng8w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts\" (UID: \"0013c2ef-f446-46a7-b0e7-fc26ca792192\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"
Sep 16 18:51:48.797861 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.797832    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"
Sep 16 18:51:48.862678 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.862651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5mp\" (UniqueName: \"kubernetes.io/projected/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-kube-api-access-gs5mp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm\" (UID: \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"
Sep 16 18:51:48.862818 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.862699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm\" (UID: \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"
Sep 16 18:51:48.862818 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.862747    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm\" (UID: \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"
Sep 16 18:51:48.863185 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.863158    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm\" (UID: \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"
Sep 16 18:51:48.863306 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.863245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm\" (UID: \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"
Sep 16 18:51:48.872678 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.872655    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5mp\" (UniqueName: \"kubernetes.io/projected/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-kube-api-access-gs5mp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm\" (UID: \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"
Sep 16 18:51:48.899093 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.899041    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"
Sep 16 18:51:48.981271 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.981243    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"]
Sep 16 18:51:48.983137 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:51:48.983108    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfc02c5_cb5c_4f96_8dde_b1061d78e944.slice/crio-9b1e83e4680cdd9139f1931cdd5a933ecdfc290897ac25b729bad7ac8d53e6f7 WatchSource:0}: Error finding container 9b1e83e4680cdd9139f1931cdd5a933ecdfc290897ac25b729bad7ac8d53e6f7: Status 404 returned error can't find the container with id 9b1e83e4680cdd9139f1931cdd5a933ecdfc290897ac25b729bad7ac8d53e6f7
Sep 16 18:51:48.996413 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:48.996393    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"
Sep 16 18:51:49.078103 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.078051    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"]
Sep 16 18:51:49.079113 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:51:49.079024    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c6ffdb_8bfd_4ba6_ad02_7b8508912eac.slice/crio-00fe4fae372999bb8d2e0179ccb03e3971b500982f8f47c07e95ac92461bb36c WatchSource:0}: Error finding container 00fe4fae372999bb8d2e0179ccb03e3971b500982f8f47c07e95ac92461bb36c: Status 404 returned error can't find the container with id 00fe4fae372999bb8d2e0179ccb03e3971b500982f8f47c07e95ac92461bb36c
Sep 16 18:51:49.174834 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.174750    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"]
Sep 16 18:51:49.202937 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:51:49.202895    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0013c2ef_f446_46a7_b0e7_fc26ca792192.slice/crio-59d37042751f0a3bd36b93d5b6249284b156856edd08a118dba688a95b984af9 WatchSource:0}: Error finding container 59d37042751f0a3bd36b93d5b6249284b156856edd08a118dba688a95b984af9: Status 404 returned error can't find the container with id 59d37042751f0a3bd36b93d5b6249284b156856edd08a118dba688a95b984af9
Sep 16 18:51:49.274140 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.274115    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"]
Sep 16 18:51:49.279124 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:51:49.279098    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b21e46_cdac_47a5_a2a1_6175ad7f4d87.slice/crio-1e5d574a8c1e5cac59e6b35a26b5704253e93725a884313da897450dadc11b0b WatchSource:0}: Error finding container 1e5d574a8c1e5cac59e6b35a26b5704253e93725a884313da897450dadc11b0b: Status 404 returned error can't find the container with id 1e5d574a8c1e5cac59e6b35a26b5704253e93725a884313da897450dadc11b0b
Sep 16 18:51:49.628916 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.628884    2561 generic.go:356] "Generic (PLEG): container finished" podID="63b21e46-cdac-47a5-a2a1-6175ad7f4d87" containerID="d8f48e14f678e706ac1fbaee3e00ec34906e61b421bb6fcd710016b5911c7c07" exitCode=0
Sep 16 18:51:49.629349 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.628954    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm" event={"ID":"63b21e46-cdac-47a5-a2a1-6175ad7f4d87","Type":"ContainerDied","Data":"d8f48e14f678e706ac1fbaee3e00ec34906e61b421bb6fcd710016b5911c7c07"}
Sep 16 18:51:49.629349 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.629000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm" event={"ID":"63b21e46-cdac-47a5-a2a1-6175ad7f4d87","Type":"ContainerStarted","Data":"1e5d574a8c1e5cac59e6b35a26b5704253e93725a884313da897450dadc11b0b"}
Sep 16 18:51:49.630661 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.630640    2561 generic.go:356] "Generic (PLEG): container finished" podID="34c6ffdb-8bfd-4ba6-ad02-7b8508912eac" containerID="7b598fda564dbe6ffe01524a5d8f9ccd70c3299503961bbe945829cda4477c87" exitCode=0
Sep 16 18:51:49.630714 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.630672    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v" event={"ID":"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac","Type":"ContainerDied","Data":"7b598fda564dbe6ffe01524a5d8f9ccd70c3299503961bbe945829cda4477c87"}
Sep 16 18:51:49.630763 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.630714    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v" event={"ID":"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac","Type":"ContainerStarted","Data":"00fe4fae372999bb8d2e0179ccb03e3971b500982f8f47c07e95ac92461bb36c"}
Sep 16 18:51:49.632508 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.632483    2561 generic.go:356] "Generic (PLEG): container finished" podID="0013c2ef-f446-46a7-b0e7-fc26ca792192" containerID="cd226089cc63c50c508f064ec82d5601b69d2869f74eb9b043d61ca9aa8b27db" exitCode=0
Sep 16 18:51:49.632623 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.632564    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts" event={"ID":"0013c2ef-f446-46a7-b0e7-fc26ca792192","Type":"ContainerDied","Data":"cd226089cc63c50c508f064ec82d5601b69d2869f74eb9b043d61ca9aa8b27db"}
Sep 16 18:51:49.632623 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.632612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts" event={"ID":"0013c2ef-f446-46a7-b0e7-fc26ca792192","Type":"ContainerStarted","Data":"59d37042751f0a3bd36b93d5b6249284b156856edd08a118dba688a95b984af9"}
Sep 16 18:51:49.634373 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.634354    2561 generic.go:356] "Generic (PLEG): container finished" podID="4dfc02c5-cb5c-4f96-8dde-b1061d78e944" containerID="ae06d50a397455d53d9723e72baeeb84033a0d267574d30a4fed5b4017e2d44b" exitCode=0
Sep 16 18:51:49.634471 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.634439    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs" event={"ID":"4dfc02c5-cb5c-4f96-8dde-b1061d78e944","Type":"ContainerDied","Data":"ae06d50a397455d53d9723e72baeeb84033a0d267574d30a4fed5b4017e2d44b"}
Sep 16 18:51:49.634471 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:49.634466    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs" event={"ID":"4dfc02c5-cb5c-4f96-8dde-b1061d78e944","Type":"ContainerStarted","Data":"9b1e83e4680cdd9139f1931cdd5a933ecdfc290897ac25b729bad7ac8d53e6f7"}
Sep 16 18:51:50.642547 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:50.642513    2561 generic.go:356] "Generic (PLEG): container finished" podID="0013c2ef-f446-46a7-b0e7-fc26ca792192" containerID="c061bd7f5874126f03feedaf835e8e45e70d831b48f22004c78be029111b8c61" exitCode=0
Sep 16 18:51:50.643008 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:50.642601    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts" event={"ID":"0013c2ef-f446-46a7-b0e7-fc26ca792192","Type":"ContainerDied","Data":"c061bd7f5874126f03feedaf835e8e45e70d831b48f22004c78be029111b8c61"}
Sep 16 18:51:50.644672 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:50.644651    2561 generic.go:356] "Generic (PLEG): container finished" podID="4dfc02c5-cb5c-4f96-8dde-b1061d78e944" containerID="2e33b2805067a02f1ef0531d73d3d2118edb63fd7335947752e14572bd6d633e" exitCode=0
Sep 16 18:51:50.644784 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:50.644732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs" event={"ID":"4dfc02c5-cb5c-4f96-8dde-b1061d78e944","Type":"ContainerDied","Data":"2e33b2805067a02f1ef0531d73d3d2118edb63fd7335947752e14572bd6d633e"}
Sep 16 18:51:50.647038 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:50.647015    2561 generic.go:356] "Generic (PLEG): container finished" podID="63b21e46-cdac-47a5-a2a1-6175ad7f4d87" containerID="2e0043d1f845d51988c7b0d6e34ebc7862131a2a67e2f99b924765cca214f58e" exitCode=0
Sep 16 18:51:50.647145 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:50.647085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm" event={"ID":"63b21e46-cdac-47a5-a2a1-6175ad7f4d87","Type":"ContainerDied","Data":"2e0043d1f845d51988c7b0d6e34ebc7862131a2a67e2f99b924765cca214f58e"}
Sep 16 18:51:50.649366 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:50.649346    2561 generic.go:356] "Generic (PLEG): container finished" podID="34c6ffdb-8bfd-4ba6-ad02-7b8508912eac" containerID="522fd0947854261e950e50e580400329d53d87a208abb87c13e242a31ee08b69" exitCode=0
Sep 16 18:51:50.649463 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:50.649383    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v" event={"ID":"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac","Type":"ContainerDied","Data":"522fd0947854261e950e50e580400329d53d87a208abb87c13e242a31ee08b69"}
Sep 16 18:51:51.656998 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:51.656957    2561 generic.go:356] "Generic (PLEG): container finished" podID="0013c2ef-f446-46a7-b0e7-fc26ca792192" containerID="dbfbbaa29d5c786d03b06642a9d448ebd511ea2d9baa4f29fddc3f79625deb99" exitCode=0
Sep 16 18:51:51.657472 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:51.657047    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts" event={"ID":"0013c2ef-f446-46a7-b0e7-fc26ca792192","Type":"ContainerDied","Data":"dbfbbaa29d5c786d03b06642a9d448ebd511ea2d9baa4f29fddc3f79625deb99"}
Sep 16 18:51:51.659114 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:51.659092    2561 generic.go:356] "Generic (PLEG): container finished" podID="4dfc02c5-cb5c-4f96-8dde-b1061d78e944" containerID="2ed59f904b72166b096e759302541e41109fa75519a7ae69c27dcdd905155b1a" exitCode=0
Sep 16 18:51:51.659206 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:51.659171    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs" event={"ID":"4dfc02c5-cb5c-4f96-8dde-b1061d78e944","Type":"ContainerDied","Data":"2ed59f904b72166b096e759302541e41109fa75519a7ae69c27dcdd905155b1a"}
Sep 16 18:51:51.661130 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:51.661113    2561 generic.go:356] "Generic (PLEG): container finished" podID="63b21e46-cdac-47a5-a2a1-6175ad7f4d87" containerID="4a2b9295c6739a5ae729f8673187768d05625ef4327dfc69c9233a433f50b870" exitCode=0
Sep 16 18:51:51.661228 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:51.661142    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm" event={"ID":"63b21e46-cdac-47a5-a2a1-6175ad7f4d87","Type":"ContainerDied","Data":"4a2b9295c6739a5ae729f8673187768d05625ef4327dfc69c9233a433f50b870"}
Sep 16 18:51:51.663236 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:51.663219    2561 generic.go:356] "Generic (PLEG): container finished" podID="34c6ffdb-8bfd-4ba6-ad02-7b8508912eac" containerID="9dbc853519ddd05aa497b2f2712a3dc94ed12c60cfe67685874d3dfeeea7c3a8" exitCode=0
Sep 16 18:51:51.663315 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:51.663270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v" event={"ID":"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac","Type":"ContainerDied","Data":"9dbc853519ddd05aa497b2f2712a3dc94ed12c60cfe67685874d3dfeeea7c3a8"}
Sep 16 18:51:52.817948 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.817916    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"
Sep 16 18:51:52.854378 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.854348    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"
Sep 16 18:51:52.883588 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.883562    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"
Sep 16 18:51:52.893657 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.893629    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"
Sep 16 18:51:52.903831 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.902931    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0013c2ef-f446-46a7-b0e7-fc26ca792192-util\") pod \"0013c2ef-f446-46a7-b0e7-fc26ca792192\" (UID: \"0013c2ef-f446-46a7-b0e7-fc26ca792192\") "
Sep 16 18:51:52.903831 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.903170    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0013c2ef-f446-46a7-b0e7-fc26ca792192-bundle\") pod \"0013c2ef-f446-46a7-b0e7-fc26ca792192\" (UID: \"0013c2ef-f446-46a7-b0e7-fc26ca792192\") "
Sep 16 18:51:52.903831 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.903230    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4dxb\" (UniqueName: \"kubernetes.io/projected/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-kube-api-access-w4dxb\") pod \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\" (UID: \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\") "
Sep 16 18:51:52.903831 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.903301    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-bundle\") pod \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\" (UID: \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\") "
Sep 16 18:51:52.903831 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.903327    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xng8w\" (UniqueName: \"kubernetes.io/projected/0013c2ef-f446-46a7-b0e7-fc26ca792192-kube-api-access-xng8w\") pod \"0013c2ef-f446-46a7-b0e7-fc26ca792192\" (UID: \"0013c2ef-f446-46a7-b0e7-fc26ca792192\") "
Sep 16 18:51:52.903831 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.903392    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-util\") pod \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\" (UID: \"4dfc02c5-cb5c-4f96-8dde-b1061d78e944\") "
Sep 16 18:51:52.905566 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.905123    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-bundle" (OuterVolumeSpecName: "bundle") pod "4dfc02c5-cb5c-4f96-8dde-b1061d78e944" (UID: "4dfc02c5-cb5c-4f96-8dde-b1061d78e944"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:52.905566 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.905532    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0013c2ef-f446-46a7-b0e7-fc26ca792192-bundle" (OuterVolumeSpecName: "bundle") pod "0013c2ef-f446-46a7-b0e7-fc26ca792192" (UID: "0013c2ef-f446-46a7-b0e7-fc26ca792192"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:52.908489 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.908459    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0013c2ef-f446-46a7-b0e7-fc26ca792192-kube-api-access-xng8w" (OuterVolumeSpecName: "kube-api-access-xng8w") pod "0013c2ef-f446-46a7-b0e7-fc26ca792192" (UID: "0013c2ef-f446-46a7-b0e7-fc26ca792192"). InnerVolumeSpecName "kube-api-access-xng8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:52.908774 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.908480    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-kube-api-access-w4dxb" (OuterVolumeSpecName: "kube-api-access-w4dxb") pod "4dfc02c5-cb5c-4f96-8dde-b1061d78e944" (UID: "4dfc02c5-cb5c-4f96-8dde-b1061d78e944"). InnerVolumeSpecName "kube-api-access-w4dxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:52.911173 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.911143    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0013c2ef-f446-46a7-b0e7-fc26ca792192-util" (OuterVolumeSpecName: "util") pod "0013c2ef-f446-46a7-b0e7-fc26ca792192" (UID: "0013c2ef-f446-46a7-b0e7-fc26ca792192"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:52.913810 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:52.913771    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-util" (OuterVolumeSpecName: "util") pod "4dfc02c5-cb5c-4f96-8dde-b1061d78e944" (UID: "4dfc02c5-cb5c-4f96-8dde-b1061d78e944"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:53.004644 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.004605    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-util\") pod \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\" (UID: \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\") "
Sep 16 18:51:53.004802 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.004674    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-bundle\") pod \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\" (UID: \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\") "
Sep 16 18:51:53.004802 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.004726    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs5mp\" (UniqueName: \"kubernetes.io/projected/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-kube-api-access-gs5mp\") pod \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\" (UID: \"63b21e46-cdac-47a5-a2a1-6175ad7f4d87\") "
Sep 16 18:51:53.004802 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.004751    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccb7m\" (UniqueName: \"kubernetes.io/projected/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-kube-api-access-ccb7m\") pod \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\" (UID: \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\") "
Sep 16 18:51:53.004944 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.004804    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-bundle\") pod \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\" (UID: \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\") "
Sep 16 18:51:53.004944 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.004860    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-util\") pod \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\" (UID: \"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac\") "
Sep 16 18:51:53.005172 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.005150    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0013c2ef-f446-46a7-b0e7-fc26ca792192-util\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:53.005273 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.005176    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0013c2ef-f446-46a7-b0e7-fc26ca792192-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:53.005273 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.005190    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4dxb\" (UniqueName: \"kubernetes.io/projected/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-kube-api-access-w4dxb\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:53.005273 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.005204    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:53.005273 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.005218    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xng8w\" (UniqueName: \"kubernetes.io/projected/0013c2ef-f446-46a7-b0e7-fc26ca792192-kube-api-access-xng8w\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:53.005273 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.005231    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dfc02c5-cb5c-4f96-8dde-b1061d78e944-util\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:53.005567 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.005357    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-bundle" (OuterVolumeSpecName: "bundle") pod "63b21e46-cdac-47a5-a2a1-6175ad7f4d87" (UID: "63b21e46-cdac-47a5-a2a1-6175ad7f4d87"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:53.005567 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.005355    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-bundle" (OuterVolumeSpecName: "bundle") pod "34c6ffdb-8bfd-4ba6-ad02-7b8508912eac" (UID: "34c6ffdb-8bfd-4ba6-ad02-7b8508912eac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:53.007081 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.007020    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-kube-api-access-ccb7m" (OuterVolumeSpecName: "kube-api-access-ccb7m") pod "34c6ffdb-8bfd-4ba6-ad02-7b8508912eac" (UID: "34c6ffdb-8bfd-4ba6-ad02-7b8508912eac"). InnerVolumeSpecName "kube-api-access-ccb7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:53.007538 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.007519    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-kube-api-access-gs5mp" (OuterVolumeSpecName: "kube-api-access-gs5mp") pod "63b21e46-cdac-47a5-a2a1-6175ad7f4d87" (UID: "63b21e46-cdac-47a5-a2a1-6175ad7f4d87"). InnerVolumeSpecName "kube-api-access-gs5mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:53.010442 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.010409    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-util" (OuterVolumeSpecName: "util") pod "63b21e46-cdac-47a5-a2a1-6175ad7f4d87" (UID: "63b21e46-cdac-47a5-a2a1-6175ad7f4d87"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:53.010802 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.010784    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-util" (OuterVolumeSpecName: "util") pod "34c6ffdb-8bfd-4ba6-ad02-7b8508912eac" (UID: "34c6ffdb-8bfd-4ba6-ad02-7b8508912eac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:53.106083 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.106013    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs5mp\" (UniqueName: \"kubernetes.io/projected/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-kube-api-access-gs5mp\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:53.106083 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.106084    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccb7m\" (UniqueName: \"kubernetes.io/projected/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-kube-api-access-ccb7m\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:53.106083 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.106095    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:53.106083 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.106103    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34c6ffdb-8bfd-4ba6-ad02-7b8508912eac-util\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:53.106379 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.106113    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-util\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:53.106379 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.106120    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b21e46-cdac-47a5-a2a1-6175ad7f4d87-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:51:53.676219 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.676184    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm"
Sep 16 18:51:53.676405 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.676176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf5xm" event={"ID":"63b21e46-cdac-47a5-a2a1-6175ad7f4d87","Type":"ContainerDied","Data":"1e5d574a8c1e5cac59e6b35a26b5704253e93725a884313da897450dadc11b0b"}
Sep 16 18:51:53.676405 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.676318    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e5d574a8c1e5cac59e6b35a26b5704253e93725a884313da897450dadc11b0b"
Sep 16 18:51:53.678401 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.678372    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v"
Sep 16 18:51:53.678401 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.678394    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wcl6v" event={"ID":"34c6ffdb-8bfd-4ba6-ad02-7b8508912eac","Type":"ContainerDied","Data":"00fe4fae372999bb8d2e0179ccb03e3971b500982f8f47c07e95ac92461bb36c"}
Sep 16 18:51:53.678588 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.678443    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00fe4fae372999bb8d2e0179ccb03e3971b500982f8f47c07e95ac92461bb36c"
Sep 16 18:51:53.680682 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.680658    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts"
Sep 16 18:51:53.680798 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.680662    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aq99ts" event={"ID":"0013c2ef-f446-46a7-b0e7-fc26ca792192","Type":"ContainerDied","Data":"59d37042751f0a3bd36b93d5b6249284b156856edd08a118dba688a95b984af9"}
Sep 16 18:51:53.680798 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.680789    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59d37042751f0a3bd36b93d5b6249284b156856edd08a118dba688a95b984af9"
Sep 16 18:51:53.682948 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.682925    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs" event={"ID":"4dfc02c5-cb5c-4f96-8dde-b1061d78e944","Type":"ContainerDied","Data":"9b1e83e4680cdd9139f1931cdd5a933ecdfc290897ac25b729bad7ac8d53e6f7"}
Sep 16 18:51:53.683097 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.682965    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b1e83e4680cdd9139f1931cdd5a933ecdfc290897ac25b729bad7ac8d53e6f7"
Sep 16 18:51:53.683097 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:53.682972    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpd5hs"
Sep 16 18:51:58.272530 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:51:58.272495    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:00.268049 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268014    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f64b96cb8-xkn95"]
Sep 16 18:52:00.268452 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268376    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0013c2ef-f446-46a7-b0e7-fc26ca792192" containerName="pull"
Sep 16 18:52:00.268452 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268387    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0013c2ef-f446-46a7-b0e7-fc26ca792192" containerName="pull"
Sep 16 18:52:00.268452 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268399    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c6ffdb-8bfd-4ba6-ad02-7b8508912eac" containerName="pull"
Sep 16 18:52:00.268452 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268404    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c6ffdb-8bfd-4ba6-ad02-7b8508912eac" containerName="pull"
Sep 16 18:52:00.268452 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268412    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0013c2ef-f446-46a7-b0e7-fc26ca792192" containerName="util"
Sep 16 18:52:00.268452 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268417    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0013c2ef-f446-46a7-b0e7-fc26ca792192" containerName="util"
Sep 16 18:52:00.268452 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268426    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63b21e46-cdac-47a5-a2a1-6175ad7f4d87" containerName="util"
Sep 16 18:52:00.268452 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268430    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="63b21e46-cdac-47a5-a2a1-6175ad7f4d87" containerName="util"
Sep 16 18:52:00.268452 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268436    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0013c2ef-f446-46a7-b0e7-fc26ca792192" containerName="extract"
Sep 16 18:52:00.268452 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268441    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0013c2ef-f446-46a7-b0e7-fc26ca792192" containerName="extract"
Sep 16 18:52:00.268452 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268447    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63b21e46-cdac-47a5-a2a1-6175ad7f4d87" containerName="extract"
Sep 16 18:52:00.268452 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268453    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="63b21e46-cdac-47a5-a2a1-6175ad7f4d87" containerName="extract"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268461    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c6ffdb-8bfd-4ba6-ad02-7b8508912eac" containerName="extract"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268468    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c6ffdb-8bfd-4ba6-ad02-7b8508912eac" containerName="extract"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268475    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dfc02c5-cb5c-4f96-8dde-b1061d78e944" containerName="pull"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268480    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dfc02c5-cb5c-4f96-8dde-b1061d78e944" containerName="pull"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268487    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c6ffdb-8bfd-4ba6-ad02-7b8508912eac" containerName="util"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268492    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c6ffdb-8bfd-4ba6-ad02-7b8508912eac" containerName="util"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268498    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dfc02c5-cb5c-4f96-8dde-b1061d78e944" containerName="extract"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268502    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dfc02c5-cb5c-4f96-8dde-b1061d78e944" containerName="extract"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268509    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dfc02c5-cb5c-4f96-8dde-b1061d78e944" containerName="util"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268513    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dfc02c5-cb5c-4f96-8dde-b1061d78e944" containerName="util"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268519    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63b21e46-cdac-47a5-a2a1-6175ad7f4d87" containerName="pull"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268523    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="63b21e46-cdac-47a5-a2a1-6175ad7f4d87" containerName="pull"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268577    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="63b21e46-cdac-47a5-a2a1-6175ad7f4d87" containerName="extract"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268584    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0013c2ef-f446-46a7-b0e7-fc26ca792192" containerName="extract"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268592    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="34c6ffdb-8bfd-4ba6-ad02-7b8508912eac" containerName="extract"
Sep 16 18:52:00.268797 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.268599    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dfc02c5-cb5c-4f96-8dde-b1061d78e944" containerName="extract"
Sep 16 18:52:00.272903 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.272878    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.341112 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.341069    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f64b96cb8-xkn95"]
Sep 16 18:52:00.366649 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.366616    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98sj4\" (UniqueName: \"kubernetes.io/projected/c3b46e44-354c-4b03-bc93-2aa334cf95ad-kube-api-access-98sj4\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.366831 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.366659    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3b46e44-354c-4b03-bc93-2aa334cf95ad-console-config\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.366831 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.366686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3b46e44-354c-4b03-bc93-2aa334cf95ad-console-oauth-config\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.366831 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.366755    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3b46e44-354c-4b03-bc93-2aa334cf95ad-console-serving-cert\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.366831 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.366787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b46e44-354c-4b03-bc93-2aa334cf95ad-service-ca\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.366831 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.366813    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b46e44-354c-4b03-bc93-2aa334cf95ad-trusted-ca-bundle\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.366992 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.366844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3b46e44-354c-4b03-bc93-2aa334cf95ad-oauth-serving-cert\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.370445 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.370410    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:00.392438 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.392401    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-kjvd7"]
Sep 16 18:52:00.395689 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.395671    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-kjvd7"
Sep 16 18:52:00.398872 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.398851    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:52:00.399217 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.399198    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5tfjd\""
Sep 16 18:52:00.399483 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.399469    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:52:00.422866 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.422833    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-kjvd7"]
Sep 16 18:52:00.468196 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.468166    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98sj4\" (UniqueName: \"kubernetes.io/projected/c3b46e44-354c-4b03-bc93-2aa334cf95ad-kube-api-access-98sj4\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.468360 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.468206    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbx9n\" (UniqueName: \"kubernetes.io/projected/ac3f1c53-ae03-45fd-b917-2c3e08951132-kube-api-access-nbx9n\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-kjvd7\" (UID: \"ac3f1c53-ae03-45fd-b917-2c3e08951132\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-kjvd7"
Sep 16 18:52:00.468360 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.468227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3b46e44-354c-4b03-bc93-2aa334cf95ad-console-config\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.468360 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.468262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3b46e44-354c-4b03-bc93-2aa334cf95ad-console-oauth-config\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.468360 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.468284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3b46e44-354c-4b03-bc93-2aa334cf95ad-console-serving-cert\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.468360 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.468298    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b46e44-354c-4b03-bc93-2aa334cf95ad-service-ca\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.468360 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.468320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b46e44-354c-4b03-bc93-2aa334cf95ad-trusted-ca-bundle\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.468360 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.468345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3b46e44-354c-4b03-bc93-2aa334cf95ad-oauth-serving-cert\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.469128 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.469083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3b46e44-354c-4b03-bc93-2aa334cf95ad-console-config\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.469277 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.469250    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3b46e44-354c-4b03-bc93-2aa334cf95ad-oauth-serving-cert\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.469348 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.469284    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b46e44-354c-4b03-bc93-2aa334cf95ad-service-ca\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.469348 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.469338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b46e44-354c-4b03-bc93-2aa334cf95ad-trusted-ca-bundle\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.470904 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.470879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3b46e44-354c-4b03-bc93-2aa334cf95ad-console-serving-cert\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.471021 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.471007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3b46e44-354c-4b03-bc93-2aa334cf95ad-console-oauth-config\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.485288 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.485261    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98sj4\" (UniqueName: \"kubernetes.io/projected/c3b46e44-354c-4b03-bc93-2aa334cf95ad-kube-api-access-98sj4\") pod \"console-6f64b96cb8-xkn95\" (UID: \"c3b46e44-354c-4b03-bc93-2aa334cf95ad\") " pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.506758 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.506724    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:00.569036 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.568942    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbx9n\" (UniqueName: \"kubernetes.io/projected/ac3f1c53-ae03-45fd-b917-2c3e08951132-kube-api-access-nbx9n\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-kjvd7\" (UID: \"ac3f1c53-ae03-45fd-b917-2c3e08951132\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-kjvd7"
Sep 16 18:52:00.579603 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.579577    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:52:00.581681 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.581663    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:00.590645 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.590623    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:52:00.600668 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.600632    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbx9n\" (UniqueName: \"kubernetes.io/projected/ac3f1c53-ae03-45fd-b917-2c3e08951132-kube-api-access-nbx9n\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-kjvd7\" (UID: \"ac3f1c53-ae03-45fd-b917-2c3e08951132\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-kjvd7"
Sep 16 18:52:00.708499 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.708472    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5tfjd\""
Sep 16 18:52:00.715933 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.715900    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-kjvd7"
Sep 16 18:52:00.877568 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.877532    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f64b96cb8-xkn95"]
Sep 16 18:52:00.880416 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:52:00.880385    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b46e44_354c_4b03_bc93_2aa334cf95ad.slice/crio-c587f64109319a8c45afdec93df16a73396fb96bae7838a6b4aaf3df95895a83 WatchSource:0}: Error finding container c587f64109319a8c45afdec93df16a73396fb96bae7838a6b4aaf3df95895a83: Status 404 returned error can't find the container with id c587f64109319a8c45afdec93df16a73396fb96bae7838a6b4aaf3df95895a83
Sep 16 18:52:00.901089 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:00.901036    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:01.033625 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:01.033589    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-kjvd7"]
Sep 16 18:52:01.035318 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:52:01.035278    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3f1c53_ae03_45fd_b917_2c3e08951132.slice/crio-7d8a730befdc51978dee27d213a222f3c6008cf86fd5f16d63495d5e7713060b WatchSource:0}: Error finding container 7d8a730befdc51978dee27d213a222f3c6008cf86fd5f16d63495d5e7713060b: Status 404 returned error can't find the container with id 7d8a730befdc51978dee27d213a222f3c6008cf86fd5f16d63495d5e7713060b
Sep 16 18:52:01.264614 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:01.264577    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:01.726710 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:01.726677    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-kjvd7" event={"ID":"ac3f1c53-ae03-45fd-b917-2c3e08951132","Type":"ContainerStarted","Data":"7d8a730befdc51978dee27d213a222f3c6008cf86fd5f16d63495d5e7713060b"}
Sep 16 18:52:01.728655 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:01.728631    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f64b96cb8-xkn95" event={"ID":"c3b46e44-354c-4b03-bc93-2aa334cf95ad","Type":"ContainerStarted","Data":"c77c132accf51d7d5c543b933da595381077d7723d374b416151909107d3383d"}
Sep 16 18:52:01.728808 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:01.728678    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f64b96cb8-xkn95" event={"ID":"c3b46e44-354c-4b03-bc93-2aa334cf95ad","Type":"ContainerStarted","Data":"c587f64109319a8c45afdec93df16a73396fb96bae7838a6b4aaf3df95895a83"}
Sep 16 18:52:01.738960 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:01.738931    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:01.759467 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:01.759412    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f64b96cb8-xkn95" podStartSLOduration=1.759390306 podStartE2EDuration="1.759390306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:01.756269896 +0000 UTC m=+421.558988123" watchObservedRunningTime="2026-09-16 18:52:01.759390306 +0000 UTC m=+421.562108526"
Sep 16 18:52:02.736672 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:02.736627    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-kjvd7" event={"ID":"ac3f1c53-ae03-45fd-b917-2c3e08951132","Type":"ContainerStarted","Data":"8510b35ba7fd973c75de72fed2406b55e3a5c56b3dc0b78ad2945d0855210247"}
Sep 16 18:52:02.737073 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:02.736806    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-kjvd7"
Sep 16 18:52:02.763471 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:02.763415    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-kjvd7" podStartSLOduration=1.45984738 podStartE2EDuration="2.763397812s" totalImagesPullingTime="1.303550432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:02.761299891 +0000 UTC m=+422.564018107" watchObservedRunningTime="2026-09-16 18:52:02.763397812 +0000 UTC m=+422.566116027"
Sep 16 18:52:04.148775 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:04.148743    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:04.526382 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:04.526349    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q"]
Sep 16 18:52:04.534427 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:04.534400    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q"
Sep 16 18:52:04.537296 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:04.537271    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zqnls\""
Sep 16 18:52:04.554554 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:04.554514    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q"]
Sep 16 18:52:04.610845 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:04.610802    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3348671-0288-42bf-ae7b-14b11b51d6cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-xvd7q\" (UID: \"c3348671-0288-42bf-ae7b-14b11b51d6cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q"
Sep 16 18:52:04.610845 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:04.610850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk9z2\" (UniqueName: \"kubernetes.io/projected/c3348671-0288-42bf-ae7b-14b11b51d6cd-kube-api-access-zk9z2\") pod \"kuadrant-operator-controller-manager-6d4cb79c-xvd7q\" (UID: \"c3348671-0288-42bf-ae7b-14b11b51d6cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q"
Sep 16 18:52:04.712422 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:04.712376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3348671-0288-42bf-ae7b-14b11b51d6cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-xvd7q\" (UID: \"c3348671-0288-42bf-ae7b-14b11b51d6cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q"
Sep 16 18:52:04.712621 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:04.712434    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk9z2\" (UniqueName: \"kubernetes.io/projected/c3348671-0288-42bf-ae7b-14b11b51d6cd-kube-api-access-zk9z2\") pod \"kuadrant-operator-controller-manager-6d4cb79c-xvd7q\" (UID: \"c3348671-0288-42bf-ae7b-14b11b51d6cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q"
Sep 16 18:52:04.712794 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:04.712770    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3348671-0288-42bf-ae7b-14b11b51d6cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-xvd7q\" (UID: \"c3348671-0288-42bf-ae7b-14b11b51d6cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q"
Sep 16 18:52:04.749356 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:04.749326    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk9z2\" (UniqueName: \"kubernetes.io/projected/c3348671-0288-42bf-ae7b-14b11b51d6cd-kube-api-access-zk9z2\") pod \"kuadrant-operator-controller-manager-6d4cb79c-xvd7q\" (UID: \"c3348671-0288-42bf-ae7b-14b11b51d6cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q"
Sep 16 18:52:04.844414 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:04.844328    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q"
Sep 16 18:52:04.882980 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:04.882941    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:05.132991 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:05.132964    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q"]
Sep 16 18:52:05.135424 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:52:05.135387    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3348671_0288_42bf_ae7b_14b11b51d6cd.slice/crio-9de358b319a2fb01658d2deb365b17e78087b3a515f107ce4dff0e27f8a00606 WatchSource:0}: Error finding container 9de358b319a2fb01658d2deb365b17e78087b3a515f107ce4dff0e27f8a00606: Status 404 returned error can't find the container with id 9de358b319a2fb01658d2deb365b17e78087b3a515f107ce4dff0e27f8a00606
Sep 16 18:52:05.166021 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:05.165992    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:05.281778 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:05.281711    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:05.753206 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:05.753168    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q" event={"ID":"c3348671-0288-42bf-ae7b-14b11b51d6cd","Type":"ContainerStarted","Data":"9de358b319a2fb01658d2deb365b17e78087b3a515f107ce4dff0e27f8a00606"}
Sep 16 18:52:06.768883 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:06.768848    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:07.892968 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:07.892934    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:08.132564 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:08.132524    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:09.665574 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:09.665537    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-x4j54"]
Sep 16 18:52:09.669093 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:09.669074    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-x4j54"
Sep 16 18:52:09.672114 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:09.672090    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 18:52:09.672253 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:09.672163    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7pkts\""
Sep 16 18:52:09.680200 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:09.679666    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-x4j54"]
Sep 16 18:52:09.763985 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:09.763947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmkv5\" (UniqueName: \"kubernetes.io/projected/079a8e33-3974-4f19-b73e-6d170e993433-kube-api-access-fmkv5\") pod \"dns-operator-controller-manager-65dc8f76f5-x4j54\" (UID: \"079a8e33-3974-4f19-b73e-6d170e993433\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-x4j54"
Sep 16 18:52:09.780033 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:09.779993    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q" event={"ID":"c3348671-0288-42bf-ae7b-14b11b51d6cd","Type":"ContainerStarted","Data":"91e91416460a054913410ff201e1d8214d79ffe7da66c7e11ed63369b9aed3d1"}
Sep 16 18:52:09.780356 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:09.780335    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q"
Sep 16 18:52:09.807719 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:09.807643    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q" podStartSLOduration=2.144386189 podStartE2EDuration="5.807626378s" totalImagesPullingTime="3.663240189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:09.806212465 +0000 UTC m=+429.608930682" watchObservedRunningTime="2026-09-16 18:52:09.807626378 +0000 UTC m=+429.610344593"
Sep 16 18:52:09.864527 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:09.864487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmkv5\" (UniqueName: \"kubernetes.io/projected/079a8e33-3974-4f19-b73e-6d170e993433-kube-api-access-fmkv5\") pod \"dns-operator-controller-manager-65dc8f76f5-x4j54\" (UID: \"079a8e33-3974-4f19-b73e-6d170e993433\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-x4j54"
Sep 16 18:52:09.876966 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:09.876935    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmkv5\" (UniqueName: \"kubernetes.io/projected/079a8e33-3974-4f19-b73e-6d170e993433-kube-api-access-fmkv5\") pod \"dns-operator-controller-manager-65dc8f76f5-x4j54\" (UID: \"079a8e33-3974-4f19-b73e-6d170e993433\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-x4j54"
Sep 16 18:52:09.981565 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:09.981529    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-x4j54"
Sep 16 18:52:10.256302 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:10.256275    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-x4j54"]
Sep 16 18:52:10.258471 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:52:10.258439    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079a8e33_3974_4f19_b73e_6d170e993433.slice/crio-9ea024e4b77b0ebe993eb3390013350c0da104f4b73d468396112fb113535034 WatchSource:0}: Error finding container 9ea024e4b77b0ebe993eb3390013350c0da104f4b73d468396112fb113535034: Status 404 returned error can't find the container with id 9ea024e4b77b0ebe993eb3390013350c0da104f4b73d468396112fb113535034
Sep 16 18:52:10.582134 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:10.582025    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:10.582134 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:10.582124    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:10.587647 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:10.587618    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:10.787044 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:10.787000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-x4j54" event={"ID":"079a8e33-3974-4f19-b73e-6d170e993433","Type":"ContainerStarted","Data":"9ea024e4b77b0ebe993eb3390013350c0da104f4b73d468396112fb113535034"}
Sep 16 18:52:10.791781 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:10.791753    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f64b96cb8-xkn95"
Sep 16 18:52:10.871851 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:10.871764    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b45b774d6-wz2fk"]
Sep 16 18:52:10.888903 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:10.888872    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:10.974563 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:10.974528    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:11.073070 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:11.073032    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:11.236370 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:11.236341    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:11.329634 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:11.329597    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:11.591977 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:11.591879    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:12.043408 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:12.043362    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:12.799993 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:12.799963    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-x4j54" event={"ID":"079a8e33-3974-4f19-b73e-6d170e993433","Type":"ContainerStarted","Data":"f5985c793cd2f7bfc2aa46006f6d10aac14fee9ef9545f0b25b0104ba9a32e85"}
Sep 16 18:52:12.800188 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:12.800024    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-x4j54"
Sep 16 18:52:12.827367 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:12.823551    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-x4j54" podStartSLOduration=2.199289364 podStartE2EDuration="3.823529794s" totalImagesPullingTime="1.62424043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:12.821151905 +0000 UTC m=+432.623870122" watchObservedRunningTime="2026-09-16 18:52:12.823529794 +0000 UTC m=+432.626248011"
Sep 16 18:52:13.743811 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:13.743776    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-kjvd7"
Sep 16 18:52:14.758244 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:14.758199    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:14.929453 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:14.929421    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:15.129554 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:15.129471    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:15.240402 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:15.240374    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:15.344785 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:15.344725    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:15.447183 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:15.447101    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:20.789451 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:20.789418    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xvd7q"
Sep 16 18:52:22.477179 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:22.477149    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:23.296095 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:23.296048    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:23.456752 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:23.456712    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:23.582502 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:23.582423    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:23.809540 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:23.809510    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-x4j54"
Sep 16 18:52:25.707939 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:25.707903    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:25.805935 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:25.805842    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:29.751226 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:29.751193    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:29.852263 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:29.852229    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:30.121181 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:30.121091    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:30.230202 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:30.230167    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:33.690562 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:33.690522    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:33.789222 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:33.789189    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:34.076255 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:34.076223    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:34.139762 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:34.139721    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:34.263820 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:34.263786    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:34.363991 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:34.363905    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:35.893580 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:35.893538    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b45b774d6-wz2fk" podUID="8e622d11-f99f-43d8-af28-88e091c685e3" containerName="console" containerID="cri-o://5e5c38db2046783f7cce56150cb43176ec02a8cdaeb0657cc4c428f1b178449c" gracePeriod=15
Sep 16 18:52:36.143968 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.143902    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:52:36.195811 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.195778    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-console-config\") pod \"8e622d11-f99f-43d8-af28-88e091c685e3\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") "
Sep 16 18:52:36.195811 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.195814    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-trusted-ca-bundle\") pod \"8e622d11-f99f-43d8-af28-88e091c685e3\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") "
Sep 16 18:52:36.196102 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.195829    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-service-ca\") pod \"8e622d11-f99f-43d8-af28-88e091c685e3\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") "
Sep 16 18:52:36.196102 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.195862    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e622d11-f99f-43d8-af28-88e091c685e3-console-serving-cert\") pod \"8e622d11-f99f-43d8-af28-88e091c685e3\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") "
Sep 16 18:52:36.196102 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.195925    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-oauth-serving-cert\") pod \"8e622d11-f99f-43d8-af28-88e091c685e3\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") "
Sep 16 18:52:36.196102 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.195949    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvg5v\" (UniqueName: \"kubernetes.io/projected/8e622d11-f99f-43d8-af28-88e091c685e3-kube-api-access-fvg5v\") pod \"8e622d11-f99f-43d8-af28-88e091c685e3\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") "
Sep 16 18:52:36.196102 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.195986    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e622d11-f99f-43d8-af28-88e091c685e3-console-oauth-config\") pod \"8e622d11-f99f-43d8-af28-88e091c685e3\" (UID: \"8e622d11-f99f-43d8-af28-88e091c685e3\") "
Sep 16 18:52:36.196412 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.196389    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8e622d11-f99f-43d8-af28-88e091c685e3" (UID: "8e622d11-f99f-43d8-af28-88e091c685e3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:36.196473 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.196428    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-console-config" (OuterVolumeSpecName: "console-config") pod "8e622d11-f99f-43d8-af28-88e091c685e3" (UID: "8e622d11-f99f-43d8-af28-88e091c685e3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:36.196473 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.196438    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-service-ca" (OuterVolumeSpecName: "service-ca") pod "8e622d11-f99f-43d8-af28-88e091c685e3" (UID: "8e622d11-f99f-43d8-af28-88e091c685e3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:36.196552 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.196499    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8e622d11-f99f-43d8-af28-88e091c685e3" (UID: "8e622d11-f99f-43d8-af28-88e091c685e3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:36.198431 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.198402    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e622d11-f99f-43d8-af28-88e091c685e3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8e622d11-f99f-43d8-af28-88e091c685e3" (UID: "8e622d11-f99f-43d8-af28-88e091c685e3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:36.198546 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.198441    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e622d11-f99f-43d8-af28-88e091c685e3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8e622d11-f99f-43d8-af28-88e091c685e3" (UID: "8e622d11-f99f-43d8-af28-88e091c685e3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:36.198546 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.198458    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e622d11-f99f-43d8-af28-88e091c685e3-kube-api-access-fvg5v" (OuterVolumeSpecName: "kube-api-access-fvg5v") pod "8e622d11-f99f-43d8-af28-88e091c685e3" (UID: "8e622d11-f99f-43d8-af28-88e091c685e3"). InnerVolumeSpecName "kube-api-access-fvg5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:36.297068 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.297026    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-oauth-serving-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:52:36.297251 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.297092    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvg5v\" (UniqueName: \"kubernetes.io/projected/8e622d11-f99f-43d8-af28-88e091c685e3-kube-api-access-fvg5v\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:52:36.297251 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.297103    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e622d11-f99f-43d8-af28-88e091c685e3-console-oauth-config\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:52:36.297251 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.297113    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-console-config\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:52:36.297251 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.297122    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-trusted-ca-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:52:36.297251 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.297130    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e622d11-f99f-43d8-af28-88e091c685e3-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:52:36.297251 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.297138    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e622d11-f99f-43d8-af28-88e091c685e3-console-serving-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:52:36.929872 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.929825    2561 generic.go:356] "Generic (PLEG): container finished" podID="8e622d11-f99f-43d8-af28-88e091c685e3" containerID="5e5c38db2046783f7cce56150cb43176ec02a8cdaeb0657cc4c428f1b178449c" exitCode=0
Sep 16 18:52:36.930316 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.929884    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b45b774d6-wz2fk"
Sep 16 18:52:36.930316 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.929919    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b45b774d6-wz2fk" event={"ID":"8e622d11-f99f-43d8-af28-88e091c685e3","Type":"ContainerDied","Data":"5e5c38db2046783f7cce56150cb43176ec02a8cdaeb0657cc4c428f1b178449c"}
Sep 16 18:52:36.930316 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.929974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b45b774d6-wz2fk" event={"ID":"8e622d11-f99f-43d8-af28-88e091c685e3","Type":"ContainerDied","Data":"964b7658953475858d0fc3c59338ee37ab000d359ab740596db9bec4a8a8f64e"}
Sep 16 18:52:36.930316 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.929995    2561 scope.go:117] "RemoveContainer" containerID="5e5c38db2046783f7cce56150cb43176ec02a8cdaeb0657cc4c428f1b178449c"
Sep 16 18:52:36.938806 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.938789    2561 scope.go:117] "RemoveContainer" containerID="5e5c38db2046783f7cce56150cb43176ec02a8cdaeb0657cc4c428f1b178449c"
Sep 16 18:52:36.939182 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:52:36.939162    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5c38db2046783f7cce56150cb43176ec02a8cdaeb0657cc4c428f1b178449c\": container with ID starting with 5e5c38db2046783f7cce56150cb43176ec02a8cdaeb0657cc4c428f1b178449c not found: ID does not exist" containerID="5e5c38db2046783f7cce56150cb43176ec02a8cdaeb0657cc4c428f1b178449c"
Sep 16 18:52:36.939252 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.939192    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5c38db2046783f7cce56150cb43176ec02a8cdaeb0657cc4c428f1b178449c"} err="failed to get container status \"5e5c38db2046783f7cce56150cb43176ec02a8cdaeb0657cc4c428f1b178449c\": rpc error: code = NotFound desc = could not find container \"5e5c38db2046783f7cce56150cb43176ec02a8cdaeb0657cc4c428f1b178449c\": container with ID starting with 5e5c38db2046783f7cce56150cb43176ec02a8cdaeb0657cc4c428f1b178449c not found: ID does not exist"
Sep 16 18:52:36.951487 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.951452    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b45b774d6-wz2fk"]
Sep 16 18:52:36.954980 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:36.954959    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b45b774d6-wz2fk"]
Sep 16 18:52:38.733931 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:38.733893    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e622d11-f99f-43d8-af28-88e091c685e3" path="/var/lib/kubelet/pods/8e622d11-f99f-43d8-af28-88e091c685e3/volumes"
Sep 16 18:52:43.075122 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:43.075086    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:43.148802 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:43.148766    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:43.279285 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:43.279248    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:43.355234 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:43.355165    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:45.367769 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:45.367740    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:45.467466 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:45.467378    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:56.170299 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.170264    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5t2dz"]
Sep 16 18:52:56.170705 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.170615    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e622d11-f99f-43d8-af28-88e091c685e3" containerName="console"
Sep 16 18:52:56.170705 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.170626    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e622d11-f99f-43d8-af28-88e091c685e3" containerName="console"
Sep 16 18:52:56.170705 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.170688    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e622d11-f99f-43d8-af28-88e091c685e3" containerName="console"
Sep 16 18:52:56.172733 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.172716    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5t2dz"
Sep 16 18:52:56.175816 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.175788    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 18:52:56.176666 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.176639    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zzcxs\""
Sep 16 18:52:56.185550 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.185516    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:56.188113 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.187789    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5t2dz"]
Sep 16 18:52:56.259232 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.259201    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5t2dz"]
Sep 16 18:52:56.285096 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.285034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9d01eab-7c9d-45e0-be43-8234b410329f-config-file\") pod \"limitador-limitador-8588749977-5t2dz\" (UID: \"b9d01eab-7c9d-45e0-be43-8234b410329f\") " pod="kuadrant-system/limitador-limitador-8588749977-5t2dz"
Sep 16 18:52:56.285258 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.285150    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5nzx\" (UniqueName: \"kubernetes.io/projected/b9d01eab-7c9d-45e0-be43-8234b410329f-kube-api-access-x5nzx\") pod \"limitador-limitador-8588749977-5t2dz\" (UID: \"b9d01eab-7c9d-45e0-be43-8234b410329f\") " pod="kuadrant-system/limitador-limitador-8588749977-5t2dz"
Sep 16 18:52:56.295433 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.295398    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:56.386484 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.386449    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9d01eab-7c9d-45e0-be43-8234b410329f-config-file\") pod \"limitador-limitador-8588749977-5t2dz\" (UID: \"b9d01eab-7c9d-45e0-be43-8234b410329f\") " pod="kuadrant-system/limitador-limitador-8588749977-5t2dz"
Sep 16 18:52:56.386676 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.386494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5nzx\" (UniqueName: \"kubernetes.io/projected/b9d01eab-7c9d-45e0-be43-8234b410329f-kube-api-access-x5nzx\") pod \"limitador-limitador-8588749977-5t2dz\" (UID: \"b9d01eab-7c9d-45e0-be43-8234b410329f\") " pod="kuadrant-system/limitador-limitador-8588749977-5t2dz"
Sep 16 18:52:56.387167 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.387142    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9d01eab-7c9d-45e0-be43-8234b410329f-config-file\") pod \"limitador-limitador-8588749977-5t2dz\" (UID: \"b9d01eab-7c9d-45e0-be43-8234b410329f\") " pod="kuadrant-system/limitador-limitador-8588749977-5t2dz"
Sep 16 18:52:56.400818 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.400762    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:56.402859 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.402836    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5nzx\" (UniqueName: \"kubernetes.io/projected/b9d01eab-7c9d-45e0-be43-8234b410329f-kube-api-access-x5nzx\") pod \"limitador-limitador-8588749977-5t2dz\" (UID: \"b9d01eab-7c9d-45e0-be43-8234b410329f\") " pod="kuadrant-system/limitador-limitador-8588749977-5t2dz"
Sep 16 18:52:56.491570 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.491531    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5t2dz"
Sep 16 18:52:56.764850 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:56.764821    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5t2dz"]
Sep 16 18:52:56.767078 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:52:56.767035    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d01eab_7c9d_45e0_be43_8234b410329f.slice/crio-ef688e2fd84a67cdb2c2a3c82a0478e224b0461c7116dad7ee11d1ffe7cc9b95 WatchSource:0}: Error finding container ef688e2fd84a67cdb2c2a3c82a0478e224b0461c7116dad7ee11d1ffe7cc9b95: Status 404 returned error can't find the container with id ef688e2fd84a67cdb2c2a3c82a0478e224b0461c7116dad7ee11d1ffe7cc9b95
Sep 16 18:52:57.003454 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:57.003423    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:57.046146 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:57.046033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5t2dz" event={"ID":"b9d01eab-7c9d-45e0-be43-8234b410329f","Type":"ContainerStarted","Data":"ef688e2fd84a67cdb2c2a3c82a0478e224b0461c7116dad7ee11d1ffe7cc9b95"}
Sep 16 18:52:57.081696 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:57.081659    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:57.083119 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:57.083098    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-spx4x"]
Sep 16 18:52:57.086288 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:57.086269    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-spx4x"
Sep 16 18:52:57.089174 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:57.089155    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pkdmw\""
Sep 16 18:52:57.094594 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:57.094573    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-spx4x"]
Sep 16 18:52:57.194485 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:57.194445    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhcdt\" (UniqueName: \"kubernetes.io/projected/3ebbb054-9fa5-4ab7-94e3-3aeea7054783-kube-api-access-nhcdt\") pod \"authorino-59c974bd48-spx4x\" (UID: \"3ebbb054-9fa5-4ab7-94e3-3aeea7054783\") " pod="kuadrant-system/authorino-59c974bd48-spx4x"
Sep 16 18:52:57.295555 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:57.295511    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhcdt\" (UniqueName: \"kubernetes.io/projected/3ebbb054-9fa5-4ab7-94e3-3aeea7054783-kube-api-access-nhcdt\") pod \"authorino-59c974bd48-spx4x\" (UID: \"3ebbb054-9fa5-4ab7-94e3-3aeea7054783\") " pod="kuadrant-system/authorino-59c974bd48-spx4x"
Sep 16 18:52:57.305137 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:57.305044    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhcdt\" (UniqueName: \"kubernetes.io/projected/3ebbb054-9fa5-4ab7-94e3-3aeea7054783-kube-api-access-nhcdt\") pod \"authorino-59c974bd48-spx4x\" (UID: \"3ebbb054-9fa5-4ab7-94e3-3aeea7054783\") " pod="kuadrant-system/authorino-59c974bd48-spx4x"
Sep 16 18:52:57.405788 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:57.405757    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-spx4x"
Sep 16 18:52:57.681551 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:57.681525    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-spx4x"]
Sep 16 18:52:57.683595 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:52:57.683568    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ebbb054_9fa5_4ab7_94e3_3aeea7054783.slice/crio-3896961093f2e0ca11b847ea5c60efe2f041d13574da0c3a489dbdb92a91000d WatchSource:0}: Error finding container 3896961093f2e0ca11b847ea5c60efe2f041d13574da0c3a489dbdb92a91000d: Status 404 returned error can't find the container with id 3896961093f2e0ca11b847ea5c60efe2f041d13574da0c3a489dbdb92a91000d
Sep 16 18:52:58.053340 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:58.053315    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-spx4x" event={"ID":"3ebbb054-9fa5-4ab7-94e3-3aeea7054783","Type":"ContainerStarted","Data":"3896961093f2e0ca11b847ea5c60efe2f041d13574da0c3a489dbdb92a91000d"}
Sep 16 18:52:59.062454 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:59.062418    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5t2dz" event={"ID":"b9d01eab-7c9d-45e0-be43-8234b410329f","Type":"ContainerStarted","Data":"8d4bfb36f35055b42ae9f20e1e43c0f1973e99044d58cde1e36a0b5244171203"}
Sep 16 18:52:59.062935 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:59.062483    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-5t2dz"
Sep 16 18:52:59.080448 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:52:59.080393    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-5t2dz" podStartSLOduration=1.8008776279999998 podStartE2EDuration="3.08037351s" totalImagesPullingTime="1.279495882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:59.079016482 +0000 UTC m=+478.881734699" watchObservedRunningTime="2026-09-16 18:52:59.08037351 +0000 UTC m=+478.883091725"
Sep 16 18:53:00.070280 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:00.070243    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-spx4x" event={"ID":"3ebbb054-9fa5-4ab7-94e3-3aeea7054783","Type":"ContainerStarted","Data":"26060b2f54a726e5395f002b8575ef4302977298e65df1acafaaaaf9eb0dae92"}
Sep 16 18:53:00.088936 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:00.088879    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-spx4x" podStartSLOduration=1.450790951 podStartE2EDuration="3.088860631s" totalImagesPullingTime="1.63806968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:00.087221354 +0000 UTC m=+479.889939570" watchObservedRunningTime="2026-09-16 18:53:00.088860631 +0000 UTC m=+479.891578847"
Sep 16 18:53:10.071847 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:10.071811    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-5t2dz"
Sep 16 18:53:12.935473 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:12.935431    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5t2dz"]
Sep 16 18:53:12.935956 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:12.935698    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-5t2dz" podUID="b9d01eab-7c9d-45e0-be43-8234b410329f" containerName="limitador" containerID="cri-o://8d4bfb36f35055b42ae9f20e1e43c0f1973e99044d58cde1e36a0b5244171203" gracePeriod=30
Sep 16 18:53:13.488500 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:13.488471    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5t2dz"
Sep 16 18:53:13.545708 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:13.545620    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5nzx\" (UniqueName: \"kubernetes.io/projected/b9d01eab-7c9d-45e0-be43-8234b410329f-kube-api-access-x5nzx\") pod \"b9d01eab-7c9d-45e0-be43-8234b410329f\" (UID: \"b9d01eab-7c9d-45e0-be43-8234b410329f\") "
Sep 16 18:53:13.545708 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:13.545683    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9d01eab-7c9d-45e0-be43-8234b410329f-config-file\") pod \"b9d01eab-7c9d-45e0-be43-8234b410329f\" (UID: \"b9d01eab-7c9d-45e0-be43-8234b410329f\") "
Sep 16 18:53:13.546029 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:13.546008    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d01eab-7c9d-45e0-be43-8234b410329f-config-file" (OuterVolumeSpecName: "config-file") pod "b9d01eab-7c9d-45e0-be43-8234b410329f" (UID: "b9d01eab-7c9d-45e0-be43-8234b410329f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:53:13.547995 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:13.547975    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d01eab-7c9d-45e0-be43-8234b410329f-kube-api-access-x5nzx" (OuterVolumeSpecName: "kube-api-access-x5nzx") pod "b9d01eab-7c9d-45e0-be43-8234b410329f" (UID: "b9d01eab-7c9d-45e0-be43-8234b410329f"). InnerVolumeSpecName "kube-api-access-x5nzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:13.646943 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:13.646901    2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9d01eab-7c9d-45e0-be43-8234b410329f-config-file\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:53:13.646943 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:13.646933    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5nzx\" (UniqueName: \"kubernetes.io/projected/b9d01eab-7c9d-45e0-be43-8234b410329f-kube-api-access-x5nzx\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:53:14.148690 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:14.148646    2561 generic.go:356] "Generic (PLEG): container finished" podID="b9d01eab-7c9d-45e0-be43-8234b410329f" containerID="8d4bfb36f35055b42ae9f20e1e43c0f1973e99044d58cde1e36a0b5244171203" exitCode=0
Sep 16 18:53:14.149192 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:14.148689    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5t2dz" event={"ID":"b9d01eab-7c9d-45e0-be43-8234b410329f","Type":"ContainerDied","Data":"8d4bfb36f35055b42ae9f20e1e43c0f1973e99044d58cde1e36a0b5244171203"}
Sep 16 18:53:14.149192 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:14.148760    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5t2dz" event={"ID":"b9d01eab-7c9d-45e0-be43-8234b410329f","Type":"ContainerDied","Data":"ef688e2fd84a67cdb2c2a3c82a0478e224b0461c7116dad7ee11d1ffe7cc9b95"}
Sep 16 18:53:14.149192 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:14.148791    2561 scope.go:117] "RemoveContainer" containerID="8d4bfb36f35055b42ae9f20e1e43c0f1973e99044d58cde1e36a0b5244171203"
Sep 16 18:53:14.149192 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:14.148700    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5t2dz"
Sep 16 18:53:14.158269 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:14.158250    2561 scope.go:117] "RemoveContainer" containerID="8d4bfb36f35055b42ae9f20e1e43c0f1973e99044d58cde1e36a0b5244171203"
Sep 16 18:53:14.158584 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:53:14.158565    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4bfb36f35055b42ae9f20e1e43c0f1973e99044d58cde1e36a0b5244171203\": container with ID starting with 8d4bfb36f35055b42ae9f20e1e43c0f1973e99044d58cde1e36a0b5244171203 not found: ID does not exist" containerID="8d4bfb36f35055b42ae9f20e1e43c0f1973e99044d58cde1e36a0b5244171203"
Sep 16 18:53:14.158630 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:14.158593    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4bfb36f35055b42ae9f20e1e43c0f1973e99044d58cde1e36a0b5244171203"} err="failed to get container status \"8d4bfb36f35055b42ae9f20e1e43c0f1973e99044d58cde1e36a0b5244171203\": rpc error: code = NotFound desc = could not find container \"8d4bfb36f35055b42ae9f20e1e43c0f1973e99044d58cde1e36a0b5244171203\": container with ID starting with 8d4bfb36f35055b42ae9f20e1e43c0f1973e99044d58cde1e36a0b5244171203 not found: ID does not exist"
Sep 16 18:53:14.171170 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:14.171141    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5t2dz"]
Sep 16 18:53:14.175120 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:14.175095    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5t2dz"]
Sep 16 18:53:14.734778 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:14.734744    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d01eab-7c9d-45e0-be43-8234b410329f" path="/var/lib/kubelet/pods/b9d01eab-7c9d-45e0-be43-8234b410329f/volumes"
Sep 16 18:53:19.121371 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:19.121335    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:19.509173 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:19.509140    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:21.924281 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:21.924236    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-929g6"]
Sep 16 18:53:21.924754 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:21.924731    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d01eab-7c9d-45e0-be43-8234b410329f" containerName="limitador"
Sep 16 18:53:21.924754 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:21.924748    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d01eab-7c9d-45e0-be43-8234b410329f" containerName="limitador"
Sep 16 18:53:21.924838 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:21.924815    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d01eab-7c9d-45e0-be43-8234b410329f" containerName="limitador"
Sep 16 18:53:21.927814 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:21.927798    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-929g6"
Sep 16 18:53:21.930387 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:21.930365    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 18:53:21.937959 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:21.937934    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-929g6"]
Sep 16 18:53:22.021435 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:22.021403    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqz42\" (UniqueName: \"kubernetes.io/projected/c30e896c-2d82-4ac1-a064-53912c1729b9-kube-api-access-sqz42\") pod \"authorino-6d9898bd99-929g6\" (UID: \"c30e896c-2d82-4ac1-a064-53912c1729b9\") " pod="kuadrant-system/authorino-6d9898bd99-929g6"
Sep 16 18:53:22.021655 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:22.021455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c30e896c-2d82-4ac1-a064-53912c1729b9-tls-cert\") pod \"authorino-6d9898bd99-929g6\" (UID: \"c30e896c-2d82-4ac1-a064-53912c1729b9\") " pod="kuadrant-system/authorino-6d9898bd99-929g6"
Sep 16 18:53:22.122180 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:22.122122    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c30e896c-2d82-4ac1-a064-53912c1729b9-tls-cert\") pod \"authorino-6d9898bd99-929g6\" (UID: \"c30e896c-2d82-4ac1-a064-53912c1729b9\") " pod="kuadrant-system/authorino-6d9898bd99-929g6"
Sep 16 18:53:22.122401 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:22.122274    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqz42\" (UniqueName: \"kubernetes.io/projected/c30e896c-2d82-4ac1-a064-53912c1729b9-kube-api-access-sqz42\") pod \"authorino-6d9898bd99-929g6\" (UID: \"c30e896c-2d82-4ac1-a064-53912c1729b9\") " pod="kuadrant-system/authorino-6d9898bd99-929g6"
Sep 16 18:53:22.124773 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:22.124751    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c30e896c-2d82-4ac1-a064-53912c1729b9-tls-cert\") pod \"authorino-6d9898bd99-929g6\" (UID: \"c30e896c-2d82-4ac1-a064-53912c1729b9\") " pod="kuadrant-system/authorino-6d9898bd99-929g6"
Sep 16 18:53:22.132396 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:22.132372    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqz42\" (UniqueName: \"kubernetes.io/projected/c30e896c-2d82-4ac1-a064-53912c1729b9-kube-api-access-sqz42\") pod \"authorino-6d9898bd99-929g6\" (UID: \"c30e896c-2d82-4ac1-a064-53912c1729b9\") " pod="kuadrant-system/authorino-6d9898bd99-929g6"
Sep 16 18:53:22.239357 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:22.239327    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-929g6"
Sep 16 18:53:22.513088 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:22.513032    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-929g6"]
Sep 16 18:53:22.514921 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:53:22.514891    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc30e896c_2d82_4ac1_a064_53912c1729b9.slice/crio-5428f02d392d0defa29a75ab00edda88d675b703e707f75dac841ec6fb3650c8 WatchSource:0}: Error finding container 5428f02d392d0defa29a75ab00edda88d675b703e707f75dac841ec6fb3650c8: Status 404 returned error can't find the container with id 5428f02d392d0defa29a75ab00edda88d675b703e707f75dac841ec6fb3650c8
Sep 16 18:53:23.198493 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:23.198459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-929g6" event={"ID":"c30e896c-2d82-4ac1-a064-53912c1729b9","Type":"ContainerStarted","Data":"c2e5988ae207c9522105f83d4eac67494935eebc3fa83f34351a4ab44a244bc0"}
Sep 16 18:53:23.198493 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:23.198496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-929g6" event={"ID":"c30e896c-2d82-4ac1-a064-53912c1729b9","Type":"ContainerStarted","Data":"5428f02d392d0defa29a75ab00edda88d675b703e707f75dac841ec6fb3650c8"}
Sep 16 18:53:23.215807 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:23.215753    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-929g6" podStartSLOduration=1.724265106 podStartE2EDuration="2.215734903s" totalImagesPullingTime="491.469797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:23.215506346 +0000 UTC m=+503.018224578" watchObservedRunningTime="2026-09-16 18:53:23.215734903 +0000 UTC m=+503.018453123"
Sep 16 18:53:23.248541 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:23.248510    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-spx4x"]
Sep 16 18:53:23.248760 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:23.248731    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-spx4x" podUID="3ebbb054-9fa5-4ab7-94e3-3aeea7054783" containerName="authorino" containerID="cri-o://26060b2f54a726e5395f002b8575ef4302977298e65df1acafaaaaf9eb0dae92" gracePeriod=30
Sep 16 18:53:23.491990 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:23.491965    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-spx4x"
Sep 16 18:53:23.534790 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:23.534700    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhcdt\" (UniqueName: \"kubernetes.io/projected/3ebbb054-9fa5-4ab7-94e3-3aeea7054783-kube-api-access-nhcdt\") pod \"3ebbb054-9fa5-4ab7-94e3-3aeea7054783\" (UID: \"3ebbb054-9fa5-4ab7-94e3-3aeea7054783\") "
Sep 16 18:53:23.537030 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:23.536994    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebbb054-9fa5-4ab7-94e3-3aeea7054783-kube-api-access-nhcdt" (OuterVolumeSpecName: "kube-api-access-nhcdt") pod "3ebbb054-9fa5-4ab7-94e3-3aeea7054783" (UID: "3ebbb054-9fa5-4ab7-94e3-3aeea7054783"). InnerVolumeSpecName "kube-api-access-nhcdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:23.635351 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:23.635307    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhcdt\" (UniqueName: \"kubernetes.io/projected/3ebbb054-9fa5-4ab7-94e3-3aeea7054783-kube-api-access-nhcdt\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:53:24.205201 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:24.205163    2561 generic.go:356] "Generic (PLEG): container finished" podID="3ebbb054-9fa5-4ab7-94e3-3aeea7054783" containerID="26060b2f54a726e5395f002b8575ef4302977298e65df1acafaaaaf9eb0dae92" exitCode=0
Sep 16 18:53:24.205201 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:24.205202    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-spx4x"
Sep 16 18:53:24.205701 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:24.205245    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-spx4x" event={"ID":"3ebbb054-9fa5-4ab7-94e3-3aeea7054783","Type":"ContainerDied","Data":"26060b2f54a726e5395f002b8575ef4302977298e65df1acafaaaaf9eb0dae92"}
Sep 16 18:53:24.205701 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:24.205280    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-spx4x" event={"ID":"3ebbb054-9fa5-4ab7-94e3-3aeea7054783","Type":"ContainerDied","Data":"3896961093f2e0ca11b847ea5c60efe2f041d13574da0c3a489dbdb92a91000d"}
Sep 16 18:53:24.205701 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:24.205297    2561 scope.go:117] "RemoveContainer" containerID="26060b2f54a726e5395f002b8575ef4302977298e65df1acafaaaaf9eb0dae92"
Sep 16 18:53:24.214702 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:24.214685    2561 scope.go:117] "RemoveContainer" containerID="26060b2f54a726e5395f002b8575ef4302977298e65df1acafaaaaf9eb0dae92"
Sep 16 18:53:24.214941 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:53:24.214923    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26060b2f54a726e5395f002b8575ef4302977298e65df1acafaaaaf9eb0dae92\": container with ID starting with 26060b2f54a726e5395f002b8575ef4302977298e65df1acafaaaaf9eb0dae92 not found: ID does not exist" containerID="26060b2f54a726e5395f002b8575ef4302977298e65df1acafaaaaf9eb0dae92"
Sep 16 18:53:24.214989 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:24.214953    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26060b2f54a726e5395f002b8575ef4302977298e65df1acafaaaaf9eb0dae92"} err="failed to get container status \"26060b2f54a726e5395f002b8575ef4302977298e65df1acafaaaaf9eb0dae92\": rpc error: code = NotFound desc = could not find container \"26060b2f54a726e5395f002b8575ef4302977298e65df1acafaaaaf9eb0dae92\": container with ID starting with 26060b2f54a726e5395f002b8575ef4302977298e65df1acafaaaaf9eb0dae92 not found: ID does not exist"
Sep 16 18:53:24.232481 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:24.232442    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-spx4x"]
Sep 16 18:53:24.237703 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:24.237669    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-spx4x"]
Sep 16 18:53:24.734403 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:24.734371    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ebbb054-9fa5-4ab7-94e3-3aeea7054783" path="/var/lib/kubelet/pods/3ebbb054-9fa5-4ab7-94e3-3aeea7054783/volumes"
Sep 16 18:53:27.365803 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:27.365769    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:27.469305 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:27.469276    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:42.093432 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.093392    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"]
Sep 16 18:53:42.093855 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.093803    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ebbb054-9fa5-4ab7-94e3-3aeea7054783" containerName="authorino"
Sep 16 18:53:42.093855 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.093814    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ebbb054-9fa5-4ab7-94e3-3aeea7054783" containerName="authorino"
Sep 16 18:53:42.093930 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.093865    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ebbb054-9fa5-4ab7-94e3-3aeea7054783" containerName="authorino"
Sep 16 18:53:42.099945 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.099921    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.109763 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.109736    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"]
Sep 16 18:53:42.219561 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.219527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.219561 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.219563    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.219798 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.219590    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.219798 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.219675    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.219798 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.219698    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.219798 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.219735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.219798 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.219758    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwbr\" (UniqueName: \"kubernetes.io/projected/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-kube-api-access-plwbr\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.320796 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.320757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.320796 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.320798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.321041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.320820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.321041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.320864    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.321041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.320889    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.321041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.320938    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.321041 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.320976    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwbr\" (UniqueName: \"kubernetes.io/projected/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-kube-api-access-plwbr\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.321992 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.321958    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.323414 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.323382    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.323600 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.323577    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.323680 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.323590    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.323723 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.323687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.333692 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.333659    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwbr\" (UniqueName: \"kubernetes.io/projected/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-kube-api-access-plwbr\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.334190 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.334171    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-w72vx\" (UID: \"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.409885 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.409789    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:42.689804 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.689773    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"]
Sep 16 18:53:42.691435 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:53:42.691413    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3a31e0_a1e9_4a8b_bcae_23ce1167ed9f.slice/crio-20c96930e18d6e04ba30d033696b8157f0a4c354f84e4d0c4d30069826016315 WatchSource:0}: Error finding container 20c96930e18d6e04ba30d033696b8157f0a4c354f84e4d0c4d30069826016315: Status 404 returned error can't find the container with id 20c96930e18d6e04ba30d033696b8157f0a4c354f84e4d0c4d30069826016315
Sep 16 18:53:42.693581 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.693548    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 18:53:42.693643 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:42.693617    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 18:53:43.316245 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:43.316200    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx" event={"ID":"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f","Type":"ContainerStarted","Data":"66cef89919fbe35c801ef3bf38f46e271ab95f83c1f826edb00ab6b549ed1541"}
Sep 16 18:53:43.316245 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:43.316249    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx" event={"ID":"0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f","Type":"ContainerStarted","Data":"20c96930e18d6e04ba30d033696b8157f0a4c354f84e4d0c4d30069826016315"}
Sep 16 18:53:43.316682 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:43.316401    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:43.318393 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:43.318368    2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-w72vx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 18:53:43.318531 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:43.318422    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx" podUID="0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:53:43.341574 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:43.341521    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx" podStartSLOduration=1.341503614 podStartE2EDuration="1.341503614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:43.338826496 +0000 UTC m=+523.141544713" watchObservedRunningTime="2026-09-16 18:53:43.341503614 +0000 UTC m=+523.144221830"
Sep 16 18:53:44.322395 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.322359    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w72vx"
Sep 16 18:53:44.400426 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.400392    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"]
Sep 16 18:53:44.400659 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.400633    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v" podUID="2ecd256b-c553-4d4a-8ed2-05216f758c01" containerName="discovery" containerID="cri-o://344e3a917d6c89e2624084b4806bc4d150fc1bc4684991f6718753d114ff7800" gracePeriod=30
Sep 16 18:53:44.549454 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.549419    2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-2r92v container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:8080/ready\": dial tcp 10.134.0.31:8080: connect: connection refused" start-of-body=
Sep 16 18:53:44.549626 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.549485    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v" podUID="2ecd256b-c553-4d4a-8ed2-05216f758c01" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.31:8080/ready\": dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 18:53:44.661898 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.661867    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:53:44.847738 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.847709    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-kubeconfig\") pod \"2ecd256b-c553-4d4a-8ed2-05216f758c01\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") "
Sep 16 18:53:44.847928 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.847754    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecd256b-c553-4d4a-8ed2-05216f758c01-local-certs\") pod \"2ecd256b-c553-4d4a-8ed2-05216f758c01\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") "
Sep 16 18:53:44.847928 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.847826    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-token\") pod \"2ecd256b-c553-4d4a-8ed2-05216f758c01\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") "
Sep 16 18:53:44.847928 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.847848    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-csr-dns-cert\") pod \"2ecd256b-c553-4d4a-8ed2-05216f758c01\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") "
Sep 16 18:53:44.847928 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.847865    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg424\" (UniqueName: \"kubernetes.io/projected/2ecd256b-c553-4d4a-8ed2-05216f758c01-kube-api-access-tg424\") pod \"2ecd256b-c553-4d4a-8ed2-05216f758c01\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") "
Sep 16 18:53:44.847928 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.847887    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-csr-ca-configmap\") pod \"2ecd256b-c553-4d4a-8ed2-05216f758c01\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") "
Sep 16 18:53:44.847928 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.847914    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-cacerts\") pod \"2ecd256b-c553-4d4a-8ed2-05216f758c01\" (UID: \"2ecd256b-c553-4d4a-8ed2-05216f758c01\") "
Sep 16 18:53:44.848494 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.848447    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "2ecd256b-c553-4d4a-8ed2-05216f758c01" (UID: "2ecd256b-c553-4d4a-8ed2-05216f758c01"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:53:44.850538 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.850516    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecd256b-c553-4d4a-8ed2-05216f758c01-local-certs" (OuterVolumeSpecName: "local-certs") pod "2ecd256b-c553-4d4a-8ed2-05216f758c01" (UID: "2ecd256b-c553-4d4a-8ed2-05216f758c01"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:44.850538 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.850522    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-cacerts" (OuterVolumeSpecName: "cacerts") pod "2ecd256b-c553-4d4a-8ed2-05216f758c01" (UID: "2ecd256b-c553-4d4a-8ed2-05216f758c01"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:53:44.850764 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.850717    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "2ecd256b-c553-4d4a-8ed2-05216f758c01" (UID: "2ecd256b-c553-4d4a-8ed2-05216f758c01"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:53:44.850764 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.850735    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecd256b-c553-4d4a-8ed2-05216f758c01-kube-api-access-tg424" (OuterVolumeSpecName: "kube-api-access-tg424") pod "2ecd256b-c553-4d4a-8ed2-05216f758c01" (UID: "2ecd256b-c553-4d4a-8ed2-05216f758c01"). InnerVolumeSpecName "kube-api-access-tg424". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:44.850853 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.850792    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ecd256b-c553-4d4a-8ed2-05216f758c01" (UID: "2ecd256b-c553-4d4a-8ed2-05216f758c01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:44.850853 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.850825    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "2ecd256b-c553-4d4a-8ed2-05216f758c01" (UID: "2ecd256b-c553-4d4a-8ed2-05216f758c01"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:53:44.949277 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.949170    2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-csr-ca-configmap\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:53:44.949277 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.949209    2561 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-cacerts\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:53:44.949277 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.949222    2561 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-kubeconfig\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:53:44.949277 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.949236    2561 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecd256b-c553-4d4a-8ed2-05216f758c01-local-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:53:44.949277 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.949248    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-token\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:53:44.949277 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.949258    2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ecd256b-c553-4d4a-8ed2-05216f758c01-istio-csr-dns-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:53:44.949277 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:44.949269    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg424\" (UniqueName: \"kubernetes.io/projected/2ecd256b-c553-4d4a-8ed2-05216f758c01-kube-api-access-tg424\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:53:45.328178 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:45.328138    2561 generic.go:356] "Generic (PLEG): container finished" podID="2ecd256b-c553-4d4a-8ed2-05216f758c01" containerID="344e3a917d6c89e2624084b4806bc4d150fc1bc4684991f6718753d114ff7800" exitCode=0
Sep 16 18:53:45.328585 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:45.328191    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"
Sep 16 18:53:45.328585 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:45.328228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v" event={"ID":"2ecd256b-c553-4d4a-8ed2-05216f758c01","Type":"ContainerDied","Data":"344e3a917d6c89e2624084b4806bc4d150fc1bc4684991f6718753d114ff7800"}
Sep 16 18:53:45.328585 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:45.328302    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v" event={"ID":"2ecd256b-c553-4d4a-8ed2-05216f758c01","Type":"ContainerDied","Data":"950a8b35c980c6b23d351d6a7827eff38ab8b07d6ee76c0432b904e0cece5db3"}
Sep 16 18:53:45.328585 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:45.328327    2561 scope.go:117] "RemoveContainer" containerID="344e3a917d6c89e2624084b4806bc4d150fc1bc4684991f6718753d114ff7800"
Sep 16 18:53:45.337908 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:45.337891    2561 scope.go:117] "RemoveContainer" containerID="344e3a917d6c89e2624084b4806bc4d150fc1bc4684991f6718753d114ff7800"
Sep 16 18:53:45.338227 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:53:45.338206    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344e3a917d6c89e2624084b4806bc4d150fc1bc4684991f6718753d114ff7800\": container with ID starting with 344e3a917d6c89e2624084b4806bc4d150fc1bc4684991f6718753d114ff7800 not found: ID does not exist" containerID="344e3a917d6c89e2624084b4806bc4d150fc1bc4684991f6718753d114ff7800"
Sep 16 18:53:45.338307 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:45.338236    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344e3a917d6c89e2624084b4806bc4d150fc1bc4684991f6718753d114ff7800"} err="failed to get container status \"344e3a917d6c89e2624084b4806bc4d150fc1bc4684991f6718753d114ff7800\": rpc error: code = NotFound desc = could not find container \"344e3a917d6c89e2624084b4806bc4d150fc1bc4684991f6718753d114ff7800\": container with ID starting with 344e3a917d6c89e2624084b4806bc4d150fc1bc4684991f6718753d114ff7800 not found: ID does not exist"
Sep 16 18:53:45.350777 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:45.350748    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"]
Sep 16 18:53:45.354717 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:45.354687    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r92v"]
Sep 16 18:53:46.734360 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:46.734326    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ecd256b-c553-4d4a-8ed2-05216f758c01" path="/var/lib/kubelet/pods/2ecd256b-c553-4d4a-8ed2-05216f758c01/volumes"
Sep 16 18:53:55.619760 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.619724    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:55.715152 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.715114    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-587fc58cc9-s9lkm"]
Sep 16 18:53:55.715489 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.715477    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ecd256b-c553-4d4a-8ed2-05216f758c01" containerName="discovery"
Sep 16 18:53:55.715532 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.715491    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ecd256b-c553-4d4a-8ed2-05216f758c01" containerName="discovery"
Sep 16 18:53:55.715573 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.715564    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ecd256b-c553-4d4a-8ed2-05216f758c01" containerName="discovery"
Sep 16 18:53:55.717682 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.717651    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:55.718819 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.718795    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-587fc58cc9-s9lkm"
Sep 16 18:53:55.721851 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.721812    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:53:55.721851 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.721836    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:53:55.722089 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.721867    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 16 18:53:55.723150 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.723129    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-rxwnk\""
Sep 16 18:53:55.728261 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.728238    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-587fc58cc9-s9lkm"]
Sep 16 18:53:55.739116 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.739087    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1321c471-8e55-4034-8d37-ef8e1936f7d4-cert\") pod \"llmisvc-controller-manager-587fc58cc9-s9lkm\" (UID: \"1321c471-8e55-4034-8d37-ef8e1936f7d4\") " pod="kserve/llmisvc-controller-manager-587fc58cc9-s9lkm"
Sep 16 18:53:55.739234 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.739215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktq82\" (UniqueName: \"kubernetes.io/projected/1321c471-8e55-4034-8d37-ef8e1936f7d4-kube-api-access-ktq82\") pod \"llmisvc-controller-manager-587fc58cc9-s9lkm\" (UID: \"1321c471-8e55-4034-8d37-ef8e1936f7d4\") " pod="kserve/llmisvc-controller-manager-587fc58cc9-s9lkm"
Sep 16 18:53:55.746163 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.746137    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-zhk5r"]
Sep 16 18:53:55.749494 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.749479    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-zhk5r"
Sep 16 18:53:55.752242 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.752224    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-k8879\""
Sep 16 18:53:55.752343 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.752258    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:53:55.756625 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.756525    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-zhk5r"]
Sep 16 18:53:55.840263 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.840229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1321c471-8e55-4034-8d37-ef8e1936f7d4-cert\") pod \"llmisvc-controller-manager-587fc58cc9-s9lkm\" (UID: \"1321c471-8e55-4034-8d37-ef8e1936f7d4\") " pod="kserve/llmisvc-controller-manager-587fc58cc9-s9lkm"
Sep 16 18:53:55.840263 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.840265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/89062e83-00cf-449d-87ad-b7e683be2bee-data\") pod \"seaweedfs-64568bcd49-zhk5r\" (UID: \"89062e83-00cf-449d-87ad-b7e683be2bee\") " pod="kserve/seaweedfs-64568bcd49-zhk5r"
Sep 16 18:53:55.840479 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.840325    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktq82\" (UniqueName: \"kubernetes.io/projected/1321c471-8e55-4034-8d37-ef8e1936f7d4-kube-api-access-ktq82\") pod \"llmisvc-controller-manager-587fc58cc9-s9lkm\" (UID: \"1321c471-8e55-4034-8d37-ef8e1936f7d4\") " pod="kserve/llmisvc-controller-manager-587fc58cc9-s9lkm"
Sep 16 18:53:55.840479 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.840346    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xq4c\" (UniqueName: \"kubernetes.io/projected/89062e83-00cf-449d-87ad-b7e683be2bee-kube-api-access-4xq4c\") pod \"seaweedfs-64568bcd49-zhk5r\" (UID: \"89062e83-00cf-449d-87ad-b7e683be2bee\") " pod="kserve/seaweedfs-64568bcd49-zhk5r"
Sep 16 18:53:55.840479 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:53:55.840394    2561 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 18:53:55.840581 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:53:55.840505    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1321c471-8e55-4034-8d37-ef8e1936f7d4-cert podName:1321c471-8e55-4034-8d37-ef8e1936f7d4 nodeName:}" failed. No retries permitted until 2026-09-16 18:53:56.340484806 +0000 UTC m=+536.143203003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1321c471-8e55-4034-8d37-ef8e1936f7d4-cert") pod "llmisvc-controller-manager-587fc58cc9-s9lkm" (UID: "1321c471-8e55-4034-8d37-ef8e1936f7d4") : secret "llmisvc-webhook-server-cert" not found
Sep 16 18:53:55.857736 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.857700    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktq82\" (UniqueName: \"kubernetes.io/projected/1321c471-8e55-4034-8d37-ef8e1936f7d4-kube-api-access-ktq82\") pod \"llmisvc-controller-manager-587fc58cc9-s9lkm\" (UID: \"1321c471-8e55-4034-8d37-ef8e1936f7d4\") " pod="kserve/llmisvc-controller-manager-587fc58cc9-s9lkm"
Sep 16 18:53:55.871907 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.871833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:55.940981 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.940943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/89062e83-00cf-449d-87ad-b7e683be2bee-data\") pod \"seaweedfs-64568bcd49-zhk5r\" (UID: \"89062e83-00cf-449d-87ad-b7e683be2bee\") " pod="kserve/seaweedfs-64568bcd49-zhk5r"
Sep 16 18:53:55.941178 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.941023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xq4c\" (UniqueName: \"kubernetes.io/projected/89062e83-00cf-449d-87ad-b7e683be2bee-kube-api-access-4xq4c\") pod \"seaweedfs-64568bcd49-zhk5r\" (UID: \"89062e83-00cf-449d-87ad-b7e683be2bee\") " pod="kserve/seaweedfs-64568bcd49-zhk5r"
Sep 16 18:53:55.941336 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.941315    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/89062e83-00cf-449d-87ad-b7e683be2bee-data\") pod \"seaweedfs-64568bcd49-zhk5r\" (UID: \"89062e83-00cf-449d-87ad-b7e683be2bee\") " pod="kserve/seaweedfs-64568bcd49-zhk5r"
Sep 16 18:53:55.950457 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:55.950426    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xq4c\" (UniqueName: \"kubernetes.io/projected/89062e83-00cf-449d-87ad-b7e683be2bee-kube-api-access-4xq4c\") pod \"seaweedfs-64568bcd49-zhk5r\" (UID: \"89062e83-00cf-449d-87ad-b7e683be2bee\") " pod="kserve/seaweedfs-64568bcd49-zhk5r"
Sep 16 18:53:56.059778 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:56.059744    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-zhk5r"
Sep 16 18:53:56.081773 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:56.081743    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:56.151421 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:56.151342    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:56.339432 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:56.339407    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-zhk5r"]
Sep 16 18:53:56.342197 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:53:56.342167    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89062e83_00cf_449d_87ad_b7e683be2bee.slice/crio-c2482466bb9e306e332a4ef96073862de8c83dbec75b8a7376518e1fb53fa516 WatchSource:0}: Error finding container c2482466bb9e306e332a4ef96073862de8c83dbec75b8a7376518e1fb53fa516: Status 404 returned error can't find the container with id c2482466bb9e306e332a4ef96073862de8c83dbec75b8a7376518e1fb53fa516
Sep 16 18:53:56.344211 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:56.344182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1321c471-8e55-4034-8d37-ef8e1936f7d4-cert\") pod \"llmisvc-controller-manager-587fc58cc9-s9lkm\" (UID: \"1321c471-8e55-4034-8d37-ef8e1936f7d4\") " pod="kserve/llmisvc-controller-manager-587fc58cc9-s9lkm"
Sep 16 18:53:56.346848 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:56.346826    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1321c471-8e55-4034-8d37-ef8e1936f7d4-cert\") pod \"llmisvc-controller-manager-587fc58cc9-s9lkm\" (UID: \"1321c471-8e55-4034-8d37-ef8e1936f7d4\") " pod="kserve/llmisvc-controller-manager-587fc58cc9-s9lkm"
Sep 16 18:53:56.392875 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:56.392838    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-zhk5r" event={"ID":"89062e83-00cf-449d-87ad-b7e683be2bee","Type":"ContainerStarted","Data":"c2482466bb9e306e332a4ef96073862de8c83dbec75b8a7376518e1fb53fa516"}
Sep 16 18:53:56.631102 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:56.631042    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-587fc58cc9-s9lkm"
Sep 16 18:53:56.759470 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:56.759428    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:56.882996 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:56.882921    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:56.909665 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:56.909634    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-587fc58cc9-s9lkm"]
Sep 16 18:53:56.949180 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:56.949083    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:56.969384 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:53:56.969345    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1321c471_8e55_4034_8d37_ef8e1936f7d4.slice/crio-06da108995116bbbf2ef410c6a764259f63d4d925bb64f4108793349a23f02b1 WatchSource:0}: Error finding container 06da108995116bbbf2ef410c6a764259f63d4d925bb64f4108793349a23f02b1: Status 404 returned error can't find the container with id 06da108995116bbbf2ef410c6a764259f63d4d925bb64f4108793349a23f02b1
Sep 16 18:53:57.402149 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:53:57.402098    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-587fc58cc9-s9lkm" event={"ID":"1321c471-8e55-4034-8d37-ef8e1936f7d4","Type":"ContainerStarted","Data":"06da108995116bbbf2ef410c6a764259f63d4d925bb64f4108793349a23f02b1"}
Sep 16 18:54:01.429127 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:54:01.429084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-587fc58cc9-s9lkm" event={"ID":"1321c471-8e55-4034-8d37-ef8e1936f7d4","Type":"ContainerStarted","Data":"0fea7009efa72e05e5b2d637b7454c9aa89e59a1bec0b7c13a8b33aea039fc2b"}
Sep 16 18:54:01.429632 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:54:01.429166    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-587fc58cc9-s9lkm"
Sep 16 18:54:01.430721 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:54:01.430701    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-zhk5r" event={"ID":"89062e83-00cf-449d-87ad-b7e683be2bee","Type":"ContainerStarted","Data":"5cd09c338b02a6488924cad864077b1a765728a3ad9bbf4e6fc40c090b7ee262"}
Sep 16 18:54:01.430827 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:54:01.430812    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-zhk5r"
Sep 16 18:54:01.449503 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:54:01.449436    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-587fc58cc9-s9lkm" podStartSLOduration=3.044229367 podStartE2EDuration="6.449415391s" totalImagesPullingTime="3.405186024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:01.447323003 +0000 UTC m=+541.250041224" watchObservedRunningTime="2026-09-16 18:54:01.449415391 +0000 UTC m=+541.252133608"
Sep 16 18:54:01.465499 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:54:01.465439    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-zhk5r" podStartSLOduration=2.380583845 podStartE2EDuration="6.465423018s" totalImagesPullingTime="4.084839173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:01.464875643 +0000 UTC m=+541.267593857" watchObservedRunningTime="2026-09-16 18:54:01.465423018 +0000 UTC m=+541.268141235"
Sep 16 18:54:07.438265 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:54:07.438221    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-zhk5r"
Sep 16 18:54:20.614637 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:54:20.614603    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:20.785463 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:54:20.785427    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:26.007591 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:54:26.007557    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:26.083274 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:54:26.083243    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:32.438786 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:54:32.438750    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-587fc58cc9-s9lkm"
Sep 16 18:54:32.698702 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:54:32.698605    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:32.759097 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:54:32.759010    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:00.659802 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:00.659769    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 18:55:00.667840 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:00.667817    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 18:55:15.114692 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.114659    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:15.143512 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.143481    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-br8b7"]
Sep 16 18:55:15.146917 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.146896    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-br8b7"
Sep 16 18:55:15.149781 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.149752    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 18:55:15.149978 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.149957    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-td58z\""
Sep 16 18:55:15.157048 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.157025    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-br8b7"]
Sep 16 18:55:15.188958 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.188928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd9604cf-01f0-4f64-9bc3-8dcc3c5def14-tls-certs\") pod \"model-serving-api-7b954ffc95-br8b7\" (UID: \"bd9604cf-01f0-4f64-9bc3-8dcc3c5def14\") " pod="kserve/model-serving-api-7b954ffc95-br8b7"
Sep 16 18:55:15.189162 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.188972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkrdb\" (UniqueName: \"kubernetes.io/projected/bd9604cf-01f0-4f64-9bc3-8dcc3c5def14-kube-api-access-pkrdb\") pod \"model-serving-api-7b954ffc95-br8b7\" (UID: \"bd9604cf-01f0-4f64-9bc3-8dcc3c5def14\") " pod="kserve/model-serving-api-7b954ffc95-br8b7"
Sep 16 18:55:15.204377 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.204346    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:15.278344 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.278315    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:15.289627 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.289601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd9604cf-01f0-4f64-9bc3-8dcc3c5def14-tls-certs\") pod \"model-serving-api-7b954ffc95-br8b7\" (UID: \"bd9604cf-01f0-4f64-9bc3-8dcc3c5def14\") " pod="kserve/model-serving-api-7b954ffc95-br8b7"
Sep 16 18:55:15.289710 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.289646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrdb\" (UniqueName: \"kubernetes.io/projected/bd9604cf-01f0-4f64-9bc3-8dcc3c5def14-kube-api-access-pkrdb\") pod \"model-serving-api-7b954ffc95-br8b7\" (UID: \"bd9604cf-01f0-4f64-9bc3-8dcc3c5def14\") " pod="kserve/model-serving-api-7b954ffc95-br8b7"
Sep 16 18:55:15.292278 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.292242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd9604cf-01f0-4f64-9bc3-8dcc3c5def14-tls-certs\") pod \"model-serving-api-7b954ffc95-br8b7\" (UID: \"bd9604cf-01f0-4f64-9bc3-8dcc3c5def14\") " pod="kserve/model-serving-api-7b954ffc95-br8b7"
Sep 16 18:55:15.298922 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.298904    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkrdb\" (UniqueName: \"kubernetes.io/projected/bd9604cf-01f0-4f64-9bc3-8dcc3c5def14-kube-api-access-pkrdb\") pod \"model-serving-api-7b954ffc95-br8b7\" (UID: \"bd9604cf-01f0-4f64-9bc3-8dcc3c5def14\") " pod="kserve/model-serving-api-7b954ffc95-br8b7"
Sep 16 18:55:15.379124 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.379011    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:15.458643 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.458606    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-br8b7"
Sep 16 18:55:15.479296 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.479268    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:15.736136 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.736088    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-br8b7"]
Sep 16 18:55:15.738380 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:55:15.738355    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9604cf_01f0_4f64_9bc3_8dcc3c5def14.slice/crio-3f8588ccae04387eb5972982280a63921dc57a0f16b2d9e6c0f546c0ee7fca97 WatchSource:0}: Error finding container 3f8588ccae04387eb5972982280a63921dc57a0f16b2d9e6c0f546c0ee7fca97: Status 404 returned error can't find the container with id 3f8588ccae04387eb5972982280a63921dc57a0f16b2d9e6c0f546c0ee7fca97
Sep 16 18:55:15.837552 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:15.837512    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-br8b7" event={"ID":"bd9604cf-01f0-4f64-9bc3-8dcc3c5def14","Type":"ContainerStarted","Data":"3f8588ccae04387eb5972982280a63921dc57a0f16b2d9e6c0f546c0ee7fca97"}
Sep 16 18:55:16.197158 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:16.197075    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:16.297578 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:16.297479    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:18.855830 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:18.855795    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-br8b7" event={"ID":"bd9604cf-01f0-4f64-9bc3-8dcc3c5def14","Type":"ContainerStarted","Data":"3ef21e5f58bf2320ca16ff3e0c55b5ed714ca9c13b45cd06a0fbab3b669ca8f1"}
Sep 16 18:55:18.856263 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:18.855858    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-br8b7"
Sep 16 18:55:18.876710 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:18.876642    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-br8b7" podStartSLOduration=1.695675732 podStartE2EDuration="3.876621115s" totalImagesPullingTime="2.180945383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:18.87285753 +0000 UTC m=+618.675575748" watchObservedRunningTime="2026-09-16 18:55:18.876621115 +0000 UTC m=+618.679339334"
Sep 16 18:55:29.865413 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:29.865333    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-br8b7"
Sep 16 18:55:31.668717 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:31.668682    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:31.730855 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:31.730821    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:50.657323 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:55:50.657265    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:49.740784 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:56:49.740732    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:49.840826 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:56:49.840787    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:59.936234 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:56:59.936133    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:02.112656 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:02.112594    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:02.343292 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:02.343254    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:10.634379 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:10.634343    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:10.903303 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:10.903212    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:11.421390 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.421347    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"]
Sep 16 18:57:11.425730 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.425694    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"
Sep 16 18:57:11.429912 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.429886    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 18:57:11.430079 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.429922    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 18:57:11.430079 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.429886    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 18:57:11.430230 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.430081    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-xgbfh\""
Sep 16 18:57:11.430294 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.430228    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-66vtl\""
Sep 16 18:57:11.440754 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.440731    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"]
Sep 16 18:57:11.484382 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.484328    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:11.574171 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.574131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b127c7-42ce-41bd-93cf-592a29aa0bbd-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr\" (UID: \"77b127c7-42ce-41bd-93cf-592a29aa0bbd\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"
Sep 16 18:57:11.574367 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.574178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmrkd\" (UniqueName: \"kubernetes.io/projected/77b127c7-42ce-41bd-93cf-592a29aa0bbd-kube-api-access-jmrkd\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr\" (UID: \"77b127c7-42ce-41bd-93cf-592a29aa0bbd\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"
Sep 16 18:57:11.675540 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.675450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b127c7-42ce-41bd-93cf-592a29aa0bbd-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr\" (UID: \"77b127c7-42ce-41bd-93cf-592a29aa0bbd\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"
Sep 16 18:57:11.675540 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.675490    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmrkd\" (UniqueName: \"kubernetes.io/projected/77b127c7-42ce-41bd-93cf-592a29aa0bbd-kube-api-access-jmrkd\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr\" (UID: \"77b127c7-42ce-41bd-93cf-592a29aa0bbd\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"
Sep 16 18:57:11.678237 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.678198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b127c7-42ce-41bd-93cf-592a29aa0bbd-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr\" (UID: \"77b127c7-42ce-41bd-93cf-592a29aa0bbd\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"
Sep 16 18:57:11.685183 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.685154    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmrkd\" (UniqueName: \"kubernetes.io/projected/77b127c7-42ce-41bd-93cf-592a29aa0bbd-kube-api-access-jmrkd\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr\" (UID: \"77b127c7-42ce-41bd-93cf-592a29aa0bbd\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"
Sep 16 18:57:11.736557 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:11.736523    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"
Sep 16 18:57:12.014925 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:12.014729    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"]
Sep 16 18:57:12.018046 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:57:12.018016    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b127c7_42ce_41bd_93cf_592a29aa0bbd.slice/crio-59092068c4bb6dcd54deb325f7dce1f56ca38a32a655395573f7eb00769698db WatchSource:0}: Error finding container 59092068c4bb6dcd54deb325f7dce1f56ca38a32a655395573f7eb00769698db: Status 404 returned error can't find the container with id 59092068c4bb6dcd54deb325f7dce1f56ca38a32a655395573f7eb00769698db
Sep 16 18:57:12.020707 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:12.020685    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:57:12.483040 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:12.482998    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr" event={"ID":"77b127c7-42ce-41bd-93cf-592a29aa0bbd","Type":"ContainerStarted","Data":"59092068c4bb6dcd54deb325f7dce1f56ca38a32a655395573f7eb00769698db"}
Sep 16 18:57:15.506216 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:15.506177    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr" event={"ID":"77b127c7-42ce-41bd-93cf-592a29aa0bbd","Type":"ContainerStarted","Data":"a71812711eb60a2dcad579d9afaa69f326cb5d765507a92dd87fc81cb20792e1"}
Sep 16 18:57:15.506627 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:15.506288    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"
Sep 16 18:57:15.527621 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:15.527565    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr" podStartSLOduration=2.089791064 podStartE2EDuration="4.527547888s" totalImagesPullingTime="2.437756824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:15.525167946 +0000 UTC m=+735.327886163" watchObservedRunningTime="2026-09-16 18:57:15.527547888 +0000 UTC m=+735.330266223"
Sep 16 18:57:24.346640 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.346608    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:24.511450 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.511411    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:24.637508 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.637415    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:24.777859 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.777829    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:57:24.781680 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.781656    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.785547 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.785510    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:57:24.785547 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.785544    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 18:57:24.785794 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.785565    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 18:57:24.785794 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.785686    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-nmghz\""
Sep 16 18:57:24.786672 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.786623    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-b74jj\""
Sep 16 18:57:24.786672 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.786621    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 18:57:24.786872 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.786721    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 18:57:24.801186 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.801156    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:57:24.801546 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.801520    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.801621 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.801573    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.801670 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.801621    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca843f6-16d6-4a07-895f-163077701c0e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.801715 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.801692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.801757 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.801722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9frt\" (UniqueName: \"kubernetes.io/projected/6ca843f6-16d6-4a07-895f-163077701c0e-kube-api-access-d9frt\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.801757 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.801750    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.801832 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.801766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca843f6-16d6-4a07-895f-163077701c0e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.801832 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.801812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.828490 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.828456    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:24.902874 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.902778    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.902874 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.902823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9frt\" (UniqueName: \"kubernetes.io/projected/6ca843f6-16d6-4a07-895f-163077701c0e-kube-api-access-d9frt\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.902874 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.902854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.903238 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.902878    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca843f6-16d6-4a07-895f-163077701c0e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.903238 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.902935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.903238 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.903031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.903238 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.903089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.903238 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.903138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca843f6-16d6-4a07-895f-163077701c0e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.903541 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.903233    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.903541 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.903373    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.903541 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.903461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.903719 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.903571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.903824 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.903798    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca843f6-16d6-4a07-895f-163077701c0e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.905419 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.905398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.905582 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.905563    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca843f6-16d6-4a07-895f-163077701c0e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:24.911826 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:24.911801    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9frt\" (UniqueName: \"kubernetes.io/projected/6ca843f6-16d6-4a07-895f-163077701c0e-kube-api-access-d9frt\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:25.095215 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:25.095182    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:25.379607 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:25.379365    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:57:25.381921 ip-10-0-129-125 kubenswrapper[2561]: W0916 18:57:25.381885    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca843f6_16d6_4a07_895f_163077701c0e.slice/crio-6f9c0dff96c210908fc1f35c23a5e33b89aad4f5fba40b6d6d128dc0f3f0a66b WatchSource:0}: Error finding container 6f9c0dff96c210908fc1f35c23a5e33b89aad4f5fba40b6d6d128dc0f3f0a66b: Status 404 returned error can't find the container with id 6f9c0dff96c210908fc1f35c23a5e33b89aad4f5fba40b6d6d128dc0f3f0a66b
Sep 16 18:57:25.564033 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:25.563998    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6ca843f6-16d6-4a07-895f-163077701c0e","Type":"ContainerStarted","Data":"6f9c0dff96c210908fc1f35c23a5e33b89aad4f5fba40b6d6d128dc0f3f0a66b"}
Sep 16 18:57:28.585647 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:28.585603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6ca843f6-16d6-4a07-895f-163077701c0e","Type":"ContainerStarted","Data":"896c53276033fdd05c146b0208cb76dd0b7503ea7191f389034d133d29d8a5bb"}
Sep 16 18:57:31.607475 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:31.607443    2561 generic.go:356] "Generic (PLEG): container finished" podID="6ca843f6-16d6-4a07-895f-163077701c0e" containerID="896c53276033fdd05c146b0208cb76dd0b7503ea7191f389034d133d29d8a5bb" exitCode=0
Sep 16 18:57:31.607911 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:31.607521    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6ca843f6-16d6-4a07-895f-163077701c0e","Type":"ContainerDied","Data":"896c53276033fdd05c146b0208cb76dd0b7503ea7191f389034d133d29d8a5bb"}
Sep 16 18:57:46.514574 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:46.514544    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"
Sep 16 18:57:55.719757 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:57:55.719706    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:00.837684 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:58:00.837640    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6ca843f6-16d6-4a07-895f-163077701c0e","Type":"ContainerStarted","Data":"a2ed671e695e9a0be3dcdc3ea457ccbae0c1b55826fac82a5f98def60c359ed3"}
Sep 16 18:58:00.862227 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:58:00.862166    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.3618279810000002 podStartE2EDuration="36.862150037s" totalImagesPullingTime="32.172757965s" totalInitContainerRuntime="2.327564091s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:00.859152163 +0000 UTC m=+780.661870383" watchObservedRunningTime="2026-09-16 18:58:00.862150037 +0000 UTC m=+780.664868252"
Sep 16 18:58:12.514550 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:58:12.514513    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:21.126273 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:58:21.126176    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"]
Sep 16 18:58:21.128547 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:58:21.128508    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:21.201448 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:58:21.201416    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:26.513828 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:58:26.513786    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr" podUID="77b127c7-42ce-41bd-93cf-592a29aa0bbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:36.150946 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:58:36.150869    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr" podUID="77b127c7-42ce-41bd-93cf-592a29aa0bbd" containerName="main" containerID="cri-o://a71812711eb60a2dcad579d9afaa69f326cb5d765507a92dd87fc81cb20792e1" gracePeriod=45
Sep 16 18:58:36.514294 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:58:36.514255    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr" podUID="77b127c7-42ce-41bd-93cf-592a29aa0bbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:46.514516 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:58:46.514471    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr" podUID="77b127c7-42ce-41bd-93cf-592a29aa0bbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:46.514982 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:58:46.514604    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"
Sep 16 18:58:56.514171 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:58:56.514120    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr" podUID="77b127c7-42ce-41bd-93cf-592a29aa0bbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:59:06.421106 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:06.421023    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"
Sep 16 18:59:06.507379 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:06.507333    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b127c7-42ce-41bd-93cf-592a29aa0bbd-tls-certs\") pod \"77b127c7-42ce-41bd-93cf-592a29aa0bbd\" (UID: \"77b127c7-42ce-41bd-93cf-592a29aa0bbd\") "
Sep 16 18:59:06.507567 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:06.507422    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmrkd\" (UniqueName: \"kubernetes.io/projected/77b127c7-42ce-41bd-93cf-592a29aa0bbd-kube-api-access-jmrkd\") pod \"77b127c7-42ce-41bd-93cf-592a29aa0bbd\" (UID: \"77b127c7-42ce-41bd-93cf-592a29aa0bbd\") "
Sep 16 18:59:06.509760 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:06.509719    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77b127c7-42ce-41bd-93cf-592a29aa0bbd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77b127c7-42ce-41bd-93cf-592a29aa0bbd" (UID: "77b127c7-42ce-41bd-93cf-592a29aa0bbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:06.509760 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:06.509742    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b127c7-42ce-41bd-93cf-592a29aa0bbd-kube-api-access-jmrkd" (OuterVolumeSpecName: "kube-api-access-jmrkd") pod "77b127c7-42ce-41bd-93cf-592a29aa0bbd" (UID: "77b127c7-42ce-41bd-93cf-592a29aa0bbd"). InnerVolumeSpecName "kube-api-access-jmrkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:06.608926 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:06.608888    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b127c7-42ce-41bd-93cf-592a29aa0bbd-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:59:06.608926 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:06.608919    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmrkd\" (UniqueName: \"kubernetes.io/projected/77b127c7-42ce-41bd-93cf-592a29aa0bbd-kube-api-access-jmrkd\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 18:59:07.036563 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:07.036485    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:07.235231 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:07.235196    2561 generic.go:356] "Generic (PLEG): container finished" podID="77b127c7-42ce-41bd-93cf-592a29aa0bbd" containerID="a71812711eb60a2dcad579d9afaa69f326cb5d765507a92dd87fc81cb20792e1" exitCode=0
Sep 16 18:59:07.235404 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:07.235260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr" event={"ID":"77b127c7-42ce-41bd-93cf-592a29aa0bbd","Type":"ContainerDied","Data":"a71812711eb60a2dcad579d9afaa69f326cb5d765507a92dd87fc81cb20792e1"}
Sep 16 18:59:07.235404 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:07.235276    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"
Sep 16 18:59:07.235404 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:07.235302    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr" event={"ID":"77b127c7-42ce-41bd-93cf-592a29aa0bbd","Type":"ContainerDied","Data":"59092068c4bb6dcd54deb325f7dce1f56ca38a32a655395573f7eb00769698db"}
Sep 16 18:59:07.235404 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:07.235322    2561 scope.go:117] "RemoveContainer" containerID="a71812711eb60a2dcad579d9afaa69f326cb5d765507a92dd87fc81cb20792e1"
Sep 16 18:59:07.244805 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:07.244786    2561 scope.go:117] "RemoveContainer" containerID="a71812711eb60a2dcad579d9afaa69f326cb5d765507a92dd87fc81cb20792e1"
Sep 16 18:59:07.245146 ip-10-0-129-125 kubenswrapper[2561]: E0916 18:59:07.245123    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a71812711eb60a2dcad579d9afaa69f326cb5d765507a92dd87fc81cb20792e1\": container with ID starting with a71812711eb60a2dcad579d9afaa69f326cb5d765507a92dd87fc81cb20792e1 not found: ID does not exist" containerID="a71812711eb60a2dcad579d9afaa69f326cb5d765507a92dd87fc81cb20792e1"
Sep 16 18:59:07.245209 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:07.245155    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71812711eb60a2dcad579d9afaa69f326cb5d765507a92dd87fc81cb20792e1"} err="failed to get container status \"a71812711eb60a2dcad579d9afaa69f326cb5d765507a92dd87fc81cb20792e1\": rpc error: code = NotFound desc = could not find container \"a71812711eb60a2dcad579d9afaa69f326cb5d765507a92dd87fc81cb20792e1\": container with ID starting with a71812711eb60a2dcad579d9afaa69f326cb5d765507a92dd87fc81cb20792e1 not found: ID does not exist"
Sep 16 18:59:07.254999 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:07.254961    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"]
Sep 16 18:59:07.258675 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:07.258650    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-975nr"]
Sep 16 18:59:08.735412 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:08.735378    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77b127c7-42ce-41bd-93cf-592a29aa0bbd" path="/var/lib/kubelet/pods/77b127c7-42ce-41bd-93cf-592a29aa0bbd/volumes"
Sep 16 18:59:11.165255 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:11.165224    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:11.264243 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:11.264212    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:16.539916 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:16.539877    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:16.747855 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:16.747820    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:19.256243 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:19.256205    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:25.232326 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:25.232291    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:25.416358 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:25.416315    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:25.541805 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:25.541722    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:34.978261 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:34.978198    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:40.235586 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:40.235536    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:40.326071 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:40.326016    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:55.461832 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:55.461768    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:55.541645 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:55.541589    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:56.407169 ip-10-0-129-125 kubenswrapper[2561]: I0916 18:59:56.407129    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:00:00.708635 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:00.708610    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:00:00.712995 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:00.712961    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:00:11.431316 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:11.431254    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="6ca843f6-16d6-4a07-895f-163077701c0e" containerName="main" containerID="cri-o://a2ed671e695e9a0be3dcdc3ea457ccbae0c1b55826fac82a5f98def60c359ed3" gracePeriod=45
Sep 16 19:00:12.124344 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.124318    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:00:12.236318 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.236283    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-home\") pod \"6ca843f6-16d6-4a07-895f-163077701c0e\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") "
Sep 16 19:00:12.236517 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.236333    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-model-cache\") pod \"6ca843f6-16d6-4a07-895f-163077701c0e\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") "
Sep 16 19:00:12.236517 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.236382    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-dshm\") pod \"6ca843f6-16d6-4a07-895f-163077701c0e\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") "
Sep 16 19:00:12.236517 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.236407    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-tmp-dir\") pod \"6ca843f6-16d6-4a07-895f-163077701c0e\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") "
Sep 16 19:00:12.236517 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.236428    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca843f6-16d6-4a07-895f-163077701c0e-tls-certs\") pod \"6ca843f6-16d6-4a07-895f-163077701c0e\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") "
Sep 16 19:00:12.236517 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.236446    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca843f6-16d6-4a07-895f-163077701c0e-cabundle-cert\") pod \"6ca843f6-16d6-4a07-895f-163077701c0e\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") "
Sep 16 19:00:12.236517 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.236491    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-kserve-provision-location\") pod \"6ca843f6-16d6-4a07-895f-163077701c0e\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") "
Sep 16 19:00:12.236517 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.236513    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9frt\" (UniqueName: \"kubernetes.io/projected/6ca843f6-16d6-4a07-895f-163077701c0e-kube-api-access-d9frt\") pod \"6ca843f6-16d6-4a07-895f-163077701c0e\" (UID: \"6ca843f6-16d6-4a07-895f-163077701c0e\") "
Sep 16 19:00:12.236887 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.236769    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ca843f6-16d6-4a07-895f-163077701c0e" (UID: "6ca843f6-16d6-4a07-895f-163077701c0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:12.237130 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.237099    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-home" (OuterVolumeSpecName: "home") pod "6ca843f6-16d6-4a07-895f-163077701c0e" (UID: "6ca843f6-16d6-4a07-895f-163077701c0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:12.237398 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.237170    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca843f6-16d6-4a07-895f-163077701c0e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ca843f6-16d6-4a07-895f-163077701c0e" (UID: "6ca843f6-16d6-4a07-895f-163077701c0e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:12.238935 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.238894    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-dshm" (OuterVolumeSpecName: "dshm") pod "6ca843f6-16d6-4a07-895f-163077701c0e" (UID: "6ca843f6-16d6-4a07-895f-163077701c0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:12.239029 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.238981    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ca843f6-16d6-4a07-895f-163077701c0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ca843f6-16d6-4a07-895f-163077701c0e" (UID: "6ca843f6-16d6-4a07-895f-163077701c0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:12.239318 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.239300    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca843f6-16d6-4a07-895f-163077701c0e-kube-api-access-d9frt" (OuterVolumeSpecName: "kube-api-access-d9frt") pod "6ca843f6-16d6-4a07-895f-163077701c0e" (UID: "6ca843f6-16d6-4a07-895f-163077701c0e"). InnerVolumeSpecName "kube-api-access-d9frt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:12.253871 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.253841    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ca843f6-16d6-4a07-895f-163077701c0e" (UID: "6ca843f6-16d6-4a07-895f-163077701c0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:12.257413 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.257391    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ca843f6-16d6-4a07-895f-163077701c0e" (UID: "6ca843f6-16d6-4a07-895f-163077701c0e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:12.337566 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.337524    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:00:12.337566 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.337558    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:00:12.337566 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.337568    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:00:12.337813 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.337593    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:00:12.337813 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.337603    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca843f6-16d6-4a07-895f-163077701c0e-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:00:12.337813 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.337611    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca843f6-16d6-4a07-895f-163077701c0e-cabundle-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:00:12.337813 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.337620    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca843f6-16d6-4a07-895f-163077701c0e-kserve-provision-location\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:00:12.337813 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.337629    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9frt\" (UniqueName: \"kubernetes.io/projected/6ca843f6-16d6-4a07-895f-163077701c0e-kube-api-access-d9frt\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:00:12.626202 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.626113    2561 generic.go:356] "Generic (PLEG): container finished" podID="6ca843f6-16d6-4a07-895f-163077701c0e" containerID="a2ed671e695e9a0be3dcdc3ea457ccbae0c1b55826fac82a5f98def60c359ed3" exitCode=0
Sep 16 19:00:12.626202 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.626165    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:00:12.626202 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.626194    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6ca843f6-16d6-4a07-895f-163077701c0e","Type":"ContainerDied","Data":"a2ed671e695e9a0be3dcdc3ea457ccbae0c1b55826fac82a5f98def60c359ed3"}
Sep 16 19:00:12.626724 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.626245    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6ca843f6-16d6-4a07-895f-163077701c0e","Type":"ContainerDied","Data":"6f9c0dff96c210908fc1f35c23a5e33b89aad4f5fba40b6d6d128dc0f3f0a66b"}
Sep 16 19:00:12.626724 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.626268    2561 scope.go:117] "RemoveContainer" containerID="a2ed671e695e9a0be3dcdc3ea457ccbae0c1b55826fac82a5f98def60c359ed3"
Sep 16 19:00:12.636931 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.636911    2561 scope.go:117] "RemoveContainer" containerID="896c53276033fdd05c146b0208cb76dd0b7503ea7191f389034d133d29d8a5bb"
Sep 16 19:00:12.645716 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.645694    2561 scope.go:117] "RemoveContainer" containerID="a2ed671e695e9a0be3dcdc3ea457ccbae0c1b55826fac82a5f98def60c359ed3"
Sep 16 19:00:12.646025 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:00:12.645999    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ed671e695e9a0be3dcdc3ea457ccbae0c1b55826fac82a5f98def60c359ed3\": container with ID starting with a2ed671e695e9a0be3dcdc3ea457ccbae0c1b55826fac82a5f98def60c359ed3 not found: ID does not exist" containerID="a2ed671e695e9a0be3dcdc3ea457ccbae0c1b55826fac82a5f98def60c359ed3"
Sep 16 19:00:12.646152 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.646033    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ed671e695e9a0be3dcdc3ea457ccbae0c1b55826fac82a5f98def60c359ed3"} err="failed to get container status \"a2ed671e695e9a0be3dcdc3ea457ccbae0c1b55826fac82a5f98def60c359ed3\": rpc error: code = NotFound desc = could not find container \"a2ed671e695e9a0be3dcdc3ea457ccbae0c1b55826fac82a5f98def60c359ed3\": container with ID starting with a2ed671e695e9a0be3dcdc3ea457ccbae0c1b55826fac82a5f98def60c359ed3 not found: ID does not exist"
Sep 16 19:00:12.646152 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.646052    2561 scope.go:117] "RemoveContainer" containerID="896c53276033fdd05c146b0208cb76dd0b7503ea7191f389034d133d29d8a5bb"
Sep 16 19:00:12.646397 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:00:12.646382    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896c53276033fdd05c146b0208cb76dd0b7503ea7191f389034d133d29d8a5bb\": container with ID starting with 896c53276033fdd05c146b0208cb76dd0b7503ea7191f389034d133d29d8a5bb not found: ID does not exist" containerID="896c53276033fdd05c146b0208cb76dd0b7503ea7191f389034d133d29d8a5bb"
Sep 16 19:00:12.646447 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.646401    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896c53276033fdd05c146b0208cb76dd0b7503ea7191f389034d133d29d8a5bb"} err="failed to get container status \"896c53276033fdd05c146b0208cb76dd0b7503ea7191f389034d133d29d8a5bb\": rpc error: code = NotFound desc = could not find container \"896c53276033fdd05c146b0208cb76dd0b7503ea7191f389034d133d29d8a5bb\": container with ID starting with 896c53276033fdd05c146b0208cb76dd0b7503ea7191f389034d133d29d8a5bb not found: ID does not exist"
Sep 16 19:00:12.650924 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.650899    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:00:12.660833 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.660810    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:00:12.734382 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:12.734348    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ca843f6-16d6-4a07-895f-163077701c0e" path="/var/lib/kubelet/pods/6ca843f6-16d6-4a07-895f-163077701c0e/volumes"
Sep 16 19:00:15.603229 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:15.603193    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:15.669201 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:15.669161    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:30.320265 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:30.320212    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:30.423810 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:30.423774    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:36.837263 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:36.837232    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:37.014698 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.014667    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:37.231923 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.231889    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"]
Sep 16 19:00:37.232309 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.232295    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77b127c7-42ce-41bd-93cf-592a29aa0bbd" containerName="main"
Sep 16 19:00:37.232391 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.232310    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="77b127c7-42ce-41bd-93cf-592a29aa0bbd" containerName="main"
Sep 16 19:00:37.232391 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.232320    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca843f6-16d6-4a07-895f-163077701c0e" containerName="storage-initializer"
Sep 16 19:00:37.232391 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.232326    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca843f6-16d6-4a07-895f-163077701c0e" containerName="storage-initializer"
Sep 16 19:00:37.232391 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.232331    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca843f6-16d6-4a07-895f-163077701c0e" containerName="main"
Sep 16 19:00:37.232391 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.232337    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca843f6-16d6-4a07-895f-163077701c0e" containerName="main"
Sep 16 19:00:37.232660 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.232397    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="77b127c7-42ce-41bd-93cf-592a29aa0bbd" containerName="main"
Sep 16 19:00:37.232660 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.232404    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ca843f6-16d6-4a07-895f-163077701c0e" containerName="main"
Sep 16 19:00:37.235848 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.235823    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"
Sep 16 19:00:37.240642 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.240617    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 19:00:37.240787 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.240645    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-qx48n\""
Sep 16 19:00:37.240787 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.240688    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 19:00:37.240787 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.240724    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 19:00:37.240787 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.240728    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-mlvl2\""
Sep 16 19:00:37.247956 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.247928    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"]
Sep 16 19:00:37.267382 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.267328    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:37.368120 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.368052    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d2547e-fdf2-445b-b9f8-5729ba29e775-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f\" (UID: \"e9d2547e-fdf2-445b-b9f8-5729ba29e775\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"
Sep 16 19:00:37.368304 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.368155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmhtm\" (UniqueName: \"kubernetes.io/projected/e9d2547e-fdf2-445b-b9f8-5729ba29e775-kube-api-access-tmhtm\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f\" (UID: \"e9d2547e-fdf2-445b-b9f8-5729ba29e775\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"
Sep 16 19:00:37.468794 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.468760    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmhtm\" (UniqueName: \"kubernetes.io/projected/e9d2547e-fdf2-445b-b9f8-5729ba29e775-kube-api-access-tmhtm\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f\" (UID: \"e9d2547e-fdf2-445b-b9f8-5729ba29e775\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"
Sep 16 19:00:37.468973 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.468871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d2547e-fdf2-445b-b9f8-5729ba29e775-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f\" (UID: \"e9d2547e-fdf2-445b-b9f8-5729ba29e775\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"
Sep 16 19:00:37.471609 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.471588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d2547e-fdf2-445b-b9f8-5729ba29e775-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f\" (UID: \"e9d2547e-fdf2-445b-b9f8-5729ba29e775\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"
Sep 16 19:00:37.477624 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.477597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmhtm\" (UniqueName: \"kubernetes.io/projected/e9d2547e-fdf2-445b-b9f8-5729ba29e775-kube-api-access-tmhtm\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f\" (UID: \"e9d2547e-fdf2-445b-b9f8-5729ba29e775\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"
Sep 16 19:00:37.549868 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.549789    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"
Sep 16 19:00:37.840530 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:37.840453    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"]
Sep 16 19:00:37.844617 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:00:37.844589    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d2547e_fdf2_445b_b9f8_5729ba29e775.slice/crio-e4cdd785380db37bc880655c9931693ddf073d8916b98aaebe5542433694fc5f WatchSource:0}: Error finding container e4cdd785380db37bc880655c9931693ddf073d8916b98aaebe5542433694fc5f: Status 404 returned error can't find the container with id e4cdd785380db37bc880655c9931693ddf073d8916b98aaebe5542433694fc5f
Sep 16 19:00:38.776119 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:38.776050    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f" event={"ID":"e9d2547e-fdf2-445b-b9f8-5729ba29e775","Type":"ContainerStarted","Data":"6f19486b405683f9ce469d0cb5800c45a97fdcd7fa0826cd5372c0115af93d01"}
Sep 16 19:00:38.776352 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:38.776230    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"
Sep 16 19:00:38.776352 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:38.776251    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f" event={"ID":"e9d2547e-fdf2-445b-b9f8-5729ba29e775","Type":"ContainerStarted","Data":"e4cdd785380db37bc880655c9931693ddf073d8916b98aaebe5542433694fc5f"}
Sep 16 19:00:38.796827 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:38.796770    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f" podStartSLOduration=1.796748875 podStartE2EDuration="1.796748875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:38.794870836 +0000 UTC m=+938.597589053" watchObservedRunningTime="2026-09-16 19:00:38.796748875 +0000 UTC m=+938.599467090"
Sep 16 19:00:43.235164 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:43.235129    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:43.304391 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:43.304359    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:45.248383 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:45.248342    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:45.552616 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:45.552526    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:45.659117 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:45.659080    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:45.755784 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:45.755748    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:46.431355 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:46.431318    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:46.532676 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:46.532637    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:49.643277 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:49.643249    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:52.189691 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:52.189664    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:54.403902 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:54.403861    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:55.454118 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:55.454045    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:55.592217 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:55.592183    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:56.101402 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:56.101312    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:56.202590 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:56.202557    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:57.219752 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:57.219691    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:57.309101 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:57.309045    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:58.746610 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:58.746578    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:58.810019 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:58.809970    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:59.017871 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:59.017736    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:59.085313 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:00:59.085242    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:00.531972 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:00.531932    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:06.312572 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:06.312529    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:07.074725 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:07.074692    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:09.789200 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:09.789169    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"
Sep 16 19:01:10.334448 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:10.334409    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:12.762193 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:12.762154    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:12.881152 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:12.881122    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:12.991933 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:12.991845    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:14.286226 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:14.286193    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:17.859027 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:17.858996    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:22.890777 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:22.890681    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:23.694974 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:23.694935    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:23.826426 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:23.826385    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:24.490073 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:24.490030    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:25.432343 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:25.432307    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:25.695595 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:25.695501    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:28.571089 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:28.571026    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:28.571514 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:28.571226    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"]
Sep 16 19:01:28.631863 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:28.631830    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:29.486784 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:29.486747    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:29.789227 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:29.789184    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f" podUID="e9d2547e-fdf2-445b-b9f8-5729ba29e775" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:30.626258 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:30.626221    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:30.725920 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:30.725888    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:39.788950 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:39.788910    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f" podUID="e9d2547e-fdf2-445b-b9f8-5729ba29e775" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:43.593355 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:43.593286    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f" podUID="e9d2547e-fdf2-445b-b9f8-5729ba29e775" containerName="main" containerID="cri-o://6f19486b405683f9ce469d0cb5800c45a97fdcd7fa0826cd5372c0115af93d01" gracePeriod=45
Sep 16 19:01:45.514536 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:45.514499    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:49.789348 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:49.789306    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f" podUID="e9d2547e-fdf2-445b-b9f8-5729ba29e775" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:49.789759 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:49.789438    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"
Sep 16 19:01:59.789250 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:01:59.789213    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f" podUID="e9d2547e-fdf2-445b-b9f8-5729ba29e775" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:09.789173 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:09.789132    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f" podUID="e9d2547e-fdf2-445b-b9f8-5729ba29e775" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:13.850279 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:13.850211    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"
Sep 16 19:02:13.956124 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:13.956039    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d2547e-fdf2-445b-b9f8-5729ba29e775-tls-certs\") pod \"e9d2547e-fdf2-445b-b9f8-5729ba29e775\" (UID: \"e9d2547e-fdf2-445b-b9f8-5729ba29e775\") "
Sep 16 19:02:13.956362 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:13.956165    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmhtm\" (UniqueName: \"kubernetes.io/projected/e9d2547e-fdf2-445b-b9f8-5729ba29e775-kube-api-access-tmhtm\") pod \"e9d2547e-fdf2-445b-b9f8-5729ba29e775\" (UID: \"e9d2547e-fdf2-445b-b9f8-5729ba29e775\") "
Sep 16 19:02:13.958486 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:13.958462    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d2547e-fdf2-445b-b9f8-5729ba29e775-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9d2547e-fdf2-445b-b9f8-5729ba29e775" (UID: "e9d2547e-fdf2-445b-b9f8-5729ba29e775"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:13.958561 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:13.958460    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d2547e-fdf2-445b-b9f8-5729ba29e775-kube-api-access-tmhtm" (OuterVolumeSpecName: "kube-api-access-tmhtm") pod "e9d2547e-fdf2-445b-b9f8-5729ba29e775" (UID: "e9d2547e-fdf2-445b-b9f8-5729ba29e775"). InnerVolumeSpecName "kube-api-access-tmhtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:14.057336 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:14.057282    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d2547e-fdf2-445b-b9f8-5729ba29e775-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:02:14.057336 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:14.057331    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmhtm\" (UniqueName: \"kubernetes.io/projected/e9d2547e-fdf2-445b-b9f8-5729ba29e775-kube-api-access-tmhtm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:02:14.327404 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:14.327364    2561 generic.go:356] "Generic (PLEG): container finished" podID="e9d2547e-fdf2-445b-b9f8-5729ba29e775" containerID="6f19486b405683f9ce469d0cb5800c45a97fdcd7fa0826cd5372c0115af93d01" exitCode=0
Sep 16 19:02:14.327404 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:14.327408    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"
Sep 16 19:02:14.327634 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:14.327440    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f" event={"ID":"e9d2547e-fdf2-445b-b9f8-5729ba29e775","Type":"ContainerDied","Data":"6f19486b405683f9ce469d0cb5800c45a97fdcd7fa0826cd5372c0115af93d01"}
Sep 16 19:02:14.327634 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:14.327516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f" event={"ID":"e9d2547e-fdf2-445b-b9f8-5729ba29e775","Type":"ContainerDied","Data":"e4cdd785380db37bc880655c9931693ddf073d8916b98aaebe5542433694fc5f"}
Sep 16 19:02:14.327634 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:14.327539    2561 scope.go:117] "RemoveContainer" containerID="6f19486b405683f9ce469d0cb5800c45a97fdcd7fa0826cd5372c0115af93d01"
Sep 16 19:02:14.337309 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:14.337291    2561 scope.go:117] "RemoveContainer" containerID="6f19486b405683f9ce469d0cb5800c45a97fdcd7fa0826cd5372c0115af93d01"
Sep 16 19:02:14.337641 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:02:14.337620    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f19486b405683f9ce469d0cb5800c45a97fdcd7fa0826cd5372c0115af93d01\": container with ID starting with 6f19486b405683f9ce469d0cb5800c45a97fdcd7fa0826cd5372c0115af93d01 not found: ID does not exist" containerID="6f19486b405683f9ce469d0cb5800c45a97fdcd7fa0826cd5372c0115af93d01"
Sep 16 19:02:14.337689 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:14.337651    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f19486b405683f9ce469d0cb5800c45a97fdcd7fa0826cd5372c0115af93d01"} err="failed to get container status \"6f19486b405683f9ce469d0cb5800c45a97fdcd7fa0826cd5372c0115af93d01\": rpc error: code = NotFound desc = could not find container \"6f19486b405683f9ce469d0cb5800c45a97fdcd7fa0826cd5372c0115af93d01\": container with ID starting with 6f19486b405683f9ce469d0cb5800c45a97fdcd7fa0826cd5372c0115af93d01 not found: ID does not exist"
Sep 16 19:02:14.352274 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:14.352243    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"]
Sep 16 19:02:14.354476 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:14.354452    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbb4m8f"]
Sep 16 19:02:14.735384 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:14.735355    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9d2547e-fdf2-445b-b9f8-5729ba29e775" path="/var/lib/kubelet/pods/e9d2547e-fdf2-445b-b9f8-5729ba29e775/volumes"
Sep 16 19:02:15.450753 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:15.450723    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:15.544680 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:15.544647    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:22.365465 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:22.365424    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:26.113443 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.113412    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:26.268883 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.268847    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"]
Sep 16 19:02:26.269365 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.269346    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d2547e-fdf2-445b-b9f8-5729ba29e775" containerName="main"
Sep 16 19:02:26.269472 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.269368    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d2547e-fdf2-445b-b9f8-5729ba29e775" containerName="main"
Sep 16 19:02:26.269472 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.269468    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d2547e-fdf2-445b-b9f8-5729ba29e775" containerName="main"
Sep 16 19:02:26.272265 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.272238    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"
Sep 16 19:02:26.276545 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.276516    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-wrbn5\""
Sep 16 19:02:26.276721 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.276528    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 19:02:26.276721 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.276531    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 19:02:26.276721 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.276532    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 19:02:26.276895 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.276805    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-dmjt9\""
Sep 16 19:02:26.282842 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.282818    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"]
Sep 16 19:02:26.324860 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.324827    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:26.373420 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.373318    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4hwc\" (UniqueName: \"kubernetes.io/projected/6fd5bb59-7e4f-44f1-974a-37ab5ff2152a-kube-api-access-k4hwc\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz\" (UID: \"6fd5bb59-7e4f-44f1-974a-37ab5ff2152a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"
Sep 16 19:02:26.373420 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.373384    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd5bb59-7e4f-44f1-974a-37ab5ff2152a-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz\" (UID: \"6fd5bb59-7e4f-44f1-974a-37ab5ff2152a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"
Sep 16 19:02:26.474842 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.474803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4hwc\" (UniqueName: \"kubernetes.io/projected/6fd5bb59-7e4f-44f1-974a-37ab5ff2152a-kube-api-access-k4hwc\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz\" (UID: \"6fd5bb59-7e4f-44f1-974a-37ab5ff2152a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"
Sep 16 19:02:26.475041 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.474862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd5bb59-7e4f-44f1-974a-37ab5ff2152a-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz\" (UID: \"6fd5bb59-7e4f-44f1-974a-37ab5ff2152a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"
Sep 16 19:02:26.477514 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.477489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd5bb59-7e4f-44f1-974a-37ab5ff2152a-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz\" (UID: \"6fd5bb59-7e4f-44f1-974a-37ab5ff2152a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"
Sep 16 19:02:26.484641 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.484590    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4hwc\" (UniqueName: \"kubernetes.io/projected/6fd5bb59-7e4f-44f1-974a-37ab5ff2152a-kube-api-access-k4hwc\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz\" (UID: \"6fd5bb59-7e4f-44f1-974a-37ab5ff2152a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"
Sep 16 19:02:26.585176 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.585126    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"
Sep 16 19:02:26.862411 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:26.862368    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"]
Sep 16 19:02:26.864502 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:02:26.864473    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd5bb59_7e4f_44f1_974a_37ab5ff2152a.slice/crio-3078b2627255008ffa6951afe8d0122d8a42cb07b4244d3f41e7d5cee6379227 WatchSource:0}: Error finding container 3078b2627255008ffa6951afe8d0122d8a42cb07b4244d3f41e7d5cee6379227: Status 404 returned error can't find the container with id 3078b2627255008ffa6951afe8d0122d8a42cb07b4244d3f41e7d5cee6379227
Sep 16 19:02:27.401408 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:27.401375    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz" event={"ID":"6fd5bb59-7e4f-44f1-974a-37ab5ff2152a","Type":"ContainerStarted","Data":"010fd0738f13f895e3d71654b878c5cbd529a7e6c3f6bac37a6cfe9de5e18a4f"}
Sep 16 19:02:27.401836 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:27.401427    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz" event={"ID":"6fd5bb59-7e4f-44f1-974a-37ab5ff2152a","Type":"ContainerStarted","Data":"3078b2627255008ffa6951afe8d0122d8a42cb07b4244d3f41e7d5cee6379227"}
Sep 16 19:02:27.401836 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:27.401483    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"
Sep 16 19:02:27.422353 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:27.422294    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz" podStartSLOduration=1.42227087 podStartE2EDuration="1.42227087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:27.421096077 +0000 UTC m=+1047.223814300" watchObservedRunningTime="2026-09-16 19:02:27.42227087 +0000 UTC m=+1047.224989089"
Sep 16 19:02:28.787464 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:28.787432    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:28.918220 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:28.918188    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:30.674156 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:30.674121    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:58.408755 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:02:58.408719    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"
Sep 16 19:03:00.307108 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:00.307070    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"]
Sep 16 19:03:00.312044 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:00.311901    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:00.403282 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:00.403242    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:08.409518 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:08.409475    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz" podUID="6fd5bb59-7e4f-44f1-974a-37ab5ff2152a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:15.328539 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:15.328494    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz" podUID="6fd5bb59-7e4f-44f1-974a-37ab5ff2152a" containerName="main" containerID="cri-o://010fd0738f13f895e3d71654b878c5cbd529a7e6c3f6bac37a6cfe9de5e18a4f" gracePeriod=45
Sep 16 19:03:18.408902 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:18.408859    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz" podUID="6fd5bb59-7e4f-44f1-974a-37ab5ff2152a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:28.408743 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:28.408704    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz" podUID="6fd5bb59-7e4f-44f1-974a-37ab5ff2152a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:28.409204 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:28.408811    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"
Sep 16 19:03:38.409374 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:38.409329    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz" podUID="6fd5bb59-7e4f-44f1-974a-37ab5ff2152a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:45.585515 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.585486    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"
Sep 16 19:03:45.722192 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.722155    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd5bb59-7e4f-44f1-974a-37ab5ff2152a-tls-certs\") pod \"6fd5bb59-7e4f-44f1-974a-37ab5ff2152a\" (UID: \"6fd5bb59-7e4f-44f1-974a-37ab5ff2152a\") "
Sep 16 19:03:45.722386 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.722214    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4hwc\" (UniqueName: \"kubernetes.io/projected/6fd5bb59-7e4f-44f1-974a-37ab5ff2152a-kube-api-access-k4hwc\") pod \"6fd5bb59-7e4f-44f1-974a-37ab5ff2152a\" (UID: \"6fd5bb59-7e4f-44f1-974a-37ab5ff2152a\") "
Sep 16 19:03:45.724623 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.724589    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd5bb59-7e4f-44f1-974a-37ab5ff2152a-kube-api-access-k4hwc" (OuterVolumeSpecName: "kube-api-access-k4hwc") pod "6fd5bb59-7e4f-44f1-974a-37ab5ff2152a" (UID: "6fd5bb59-7e4f-44f1-974a-37ab5ff2152a"). InnerVolumeSpecName "kube-api-access-k4hwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:45.724623 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.724611    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fd5bb59-7e4f-44f1-974a-37ab5ff2152a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fd5bb59-7e4f-44f1-974a-37ab5ff2152a" (UID: "6fd5bb59-7e4f-44f1-974a-37ab5ff2152a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:45.823352 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.823305    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4hwc\" (UniqueName: \"kubernetes.io/projected/6fd5bb59-7e4f-44f1-974a-37ab5ff2152a-kube-api-access-k4hwc\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:03:45.823352 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.823343    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd5bb59-7e4f-44f1-974a-37ab5ff2152a-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:03:45.850051 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.850017    2561 generic.go:356] "Generic (PLEG): container finished" podID="6fd5bb59-7e4f-44f1-974a-37ab5ff2152a" containerID="010fd0738f13f895e3d71654b878c5cbd529a7e6c3f6bac37a6cfe9de5e18a4f" exitCode=0
Sep 16 19:03:45.850228 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.850090    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"
Sep 16 19:03:45.850228 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.850111    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz" event={"ID":"6fd5bb59-7e4f-44f1-974a-37ab5ff2152a","Type":"ContainerDied","Data":"010fd0738f13f895e3d71654b878c5cbd529a7e6c3f6bac37a6cfe9de5e18a4f"}
Sep 16 19:03:45.850228 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.850176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz" event={"ID":"6fd5bb59-7e4f-44f1-974a-37ab5ff2152a","Type":"ContainerDied","Data":"3078b2627255008ffa6951afe8d0122d8a42cb07b4244d3f41e7d5cee6379227"}
Sep 16 19:03:45.850228 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.850207    2561 scope.go:117] "RemoveContainer" containerID="010fd0738f13f895e3d71654b878c5cbd529a7e6c3f6bac37a6cfe9de5e18a4f"
Sep 16 19:03:45.860434 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.860413    2561 scope.go:117] "RemoveContainer" containerID="010fd0738f13f895e3d71654b878c5cbd529a7e6c3f6bac37a6cfe9de5e18a4f"
Sep 16 19:03:45.860773 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:03:45.860753    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010fd0738f13f895e3d71654b878c5cbd529a7e6c3f6bac37a6cfe9de5e18a4f\": container with ID starting with 010fd0738f13f895e3d71654b878c5cbd529a7e6c3f6bac37a6cfe9de5e18a4f not found: ID does not exist" containerID="010fd0738f13f895e3d71654b878c5cbd529a7e6c3f6bac37a6cfe9de5e18a4f"
Sep 16 19:03:45.860835 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.860784    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010fd0738f13f895e3d71654b878c5cbd529a7e6c3f6bac37a6cfe9de5e18a4f"} err="failed to get container status \"010fd0738f13f895e3d71654b878c5cbd529a7e6c3f6bac37a6cfe9de5e18a4f\": rpc error: code = NotFound desc = could not find container \"010fd0738f13f895e3d71654b878c5cbd529a7e6c3f6bac37a6cfe9de5e18a4f\": container with ID starting with 010fd0738f13f895e3d71654b878c5cbd529a7e6c3f6bac37a6cfe9de5e18a4f not found: ID does not exist"
Sep 16 19:03:45.873946 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.873913    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"]
Sep 16 19:03:45.880573 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:45.880550    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche86hmz"]
Sep 16 19:03:46.736069 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:46.736021    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fd5bb59-7e4f-44f1-974a-37ab5ff2152a" path="/var/lib/kubelet/pods/6fd5bb59-7e4f-44f1-974a-37ab5ff2152a/volumes"
Sep 16 19:03:50.196091 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:50.196036    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:50.305844 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:50.305815    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:55.851102 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:55.851069    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:56.062175 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.062140    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"]
Sep 16 19:03:56.062631 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.062613    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fd5bb59-7e4f-44f1-974a-37ab5ff2152a" containerName="main"
Sep 16 19:03:56.062724 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.062634    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fd5bb59-7e4f-44f1-974a-37ab5ff2152a" containerName="main"
Sep 16 19:03:56.062780 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.062750    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fd5bb59-7e4f-44f1-974a-37ab5ff2152a" containerName="main"
Sep 16 19:03:56.067343 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.067318    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"
Sep 16 19:03:56.071855 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.071828    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-zcs5m\""
Sep 16 19:03:56.071996 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.071892    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-z27bc\""
Sep 16 19:03:56.071996 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.071915    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 19:03:56.071996 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.071987    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 19:03:56.072187 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.071896    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 19:03:56.079304 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.079277    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"]
Sep 16 19:03:56.095392 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.095366    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:56.100542 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.100518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/475a82f9-a998-440b-b0c5-8885dc7f2f07-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr\" (UID: \"475a82f9-a998-440b-b0c5-8885dc7f2f07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"
Sep 16 19:03:56.100661 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.100551    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qqt8\" (UniqueName: \"kubernetes.io/projected/475a82f9-a998-440b-b0c5-8885dc7f2f07-kube-api-access-8qqt8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr\" (UID: \"475a82f9-a998-440b-b0c5-8885dc7f2f07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"
Sep 16 19:03:56.201202 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.201117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/475a82f9-a998-440b-b0c5-8885dc7f2f07-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr\" (UID: \"475a82f9-a998-440b-b0c5-8885dc7f2f07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"
Sep 16 19:03:56.201202 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.201156    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qqt8\" (UniqueName: \"kubernetes.io/projected/475a82f9-a998-440b-b0c5-8885dc7f2f07-kube-api-access-8qqt8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr\" (UID: \"475a82f9-a998-440b-b0c5-8885dc7f2f07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"
Sep 16 19:03:56.203791 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.203760    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/475a82f9-a998-440b-b0c5-8885dc7f2f07-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr\" (UID: \"475a82f9-a998-440b-b0c5-8885dc7f2f07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"
Sep 16 19:03:56.210672 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.210645    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qqt8\" (UniqueName: \"kubernetes.io/projected/475a82f9-a998-440b-b0c5-8885dc7f2f07-kube-api-access-8qqt8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr\" (UID: \"475a82f9-a998-440b-b0c5-8885dc7f2f07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"
Sep 16 19:03:56.383995 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.383947    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"
Sep 16 19:03:56.663034 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.663006    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"]
Sep 16 19:03:56.665761 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:03:56.665723    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475a82f9_a998_440b_b0c5_8885dc7f2f07.slice/crio-790e32c0d73dad05d69188fa9fa5b8e316d02068c352dcf3df34fd0cdf0f3859 WatchSource:0}: Error finding container 790e32c0d73dad05d69188fa9fa5b8e316d02068c352dcf3df34fd0cdf0f3859: Status 404 returned error can't find the container with id 790e32c0d73dad05d69188fa9fa5b8e316d02068c352dcf3df34fd0cdf0f3859
Sep 16 19:03:56.919112 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.918985    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr" event={"ID":"475a82f9-a998-440b-b0c5-8885dc7f2f07","Type":"ContainerStarted","Data":"94729f9ba718870d628d3304ed83f8301d4aa333bc6358aa6e7b7c8ab50bf877"}
Sep 16 19:03:56.919112 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.919050    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr" event={"ID":"475a82f9-a998-440b-b0c5-8885dc7f2f07","Type":"ContainerStarted","Data":"790e32c0d73dad05d69188fa9fa5b8e316d02068c352dcf3df34fd0cdf0f3859"}
Sep 16 19:03:56.919112 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.919103    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"
Sep 16 19:03:56.947575 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:56.947508    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr" podStartSLOduration=0.947489296 podStartE2EDuration="947.489296ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:56.946968963 +0000 UTC m=+1136.749687181" watchObservedRunningTime="2026-09-16 19:03:56.947489296 +0000 UTC m=+1136.750207512"
Sep 16 19:03:58.451304 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:03:58.451268    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:04.207701 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:04.207656    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:04.342261 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:04.342226    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:04.762669 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:04.762636    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:04.911638 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:04.911604    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:10.525436 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:10.525390    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:10.597442 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:10.597403    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:27.926426 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:27.926342    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"
Sep 16 19:04:29.805983 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:29.805948    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:29.835117 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:29.835083    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"]
Sep 16 19:04:29.904571 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:29.904536    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:37.925767 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:37.925727    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr" podUID="475a82f9-a998-440b-b0c5-8885dc7f2f07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:44.855844 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:44.855785    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr" podUID="475a82f9-a998-440b-b0c5-8885dc7f2f07" containerName="main" containerID="cri-o://94729f9ba718870d628d3304ed83f8301d4aa333bc6358aa6e7b7c8ab50bf877" gracePeriod=45
Sep 16 19:04:47.925883 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:47.925843    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr" podUID="475a82f9-a998-440b-b0c5-8885dc7f2f07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:57.926100 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:57.926032    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr" podUID="475a82f9-a998-440b-b0c5-8885dc7f2f07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:57.926569 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:04:57.926186    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"
Sep 16 19:05:00.614777 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:00.614741    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:00.711526 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:00.711490    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:00.754709 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:00.754683    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:05:00.756649 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:00.756624    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:05:06.949047 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:06.949006    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:07.694911 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:07.694878    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:07.921981 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:07.921929    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:07.926035 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:07.926000    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr" podUID="475a82f9-a998-440b-b0c5-8885dc7f2f07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:12.967444 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:12.967411    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:13.069418 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:13.069379    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:15.109466 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.109442    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"
Sep 16 19:05:15.226575 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.226545    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qqt8\" (UniqueName: \"kubernetes.io/projected/475a82f9-a998-440b-b0c5-8885dc7f2f07-kube-api-access-8qqt8\") pod \"475a82f9-a998-440b-b0c5-8885dc7f2f07\" (UID: \"475a82f9-a998-440b-b0c5-8885dc7f2f07\") "
Sep 16 19:05:15.226761 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.226656    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/475a82f9-a998-440b-b0c5-8885dc7f2f07-tls-certs\") pod \"475a82f9-a998-440b-b0c5-8885dc7f2f07\" (UID: \"475a82f9-a998-440b-b0c5-8885dc7f2f07\") "
Sep 16 19:05:15.229069 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.229024    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/475a82f9-a998-440b-b0c5-8885dc7f2f07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "475a82f9-a998-440b-b0c5-8885dc7f2f07" (UID: "475a82f9-a998-440b-b0c5-8885dc7f2f07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:15.229069 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.229030    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475a82f9-a998-440b-b0c5-8885dc7f2f07-kube-api-access-8qqt8" (OuterVolumeSpecName: "kube-api-access-8qqt8") pod "475a82f9-a998-440b-b0c5-8885dc7f2f07" (UID: "475a82f9-a998-440b-b0c5-8885dc7f2f07"). InnerVolumeSpecName "kube-api-access-8qqt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:15.327870 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.327827    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/475a82f9-a998-440b-b0c5-8885dc7f2f07-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:05:15.327870 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.327862    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qqt8\" (UniqueName: \"kubernetes.io/projected/475a82f9-a998-440b-b0c5-8885dc7f2f07-kube-api-access-8qqt8\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:05:15.354680 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.354641    2561 generic.go:356] "Generic (PLEG): container finished" podID="475a82f9-a998-440b-b0c5-8885dc7f2f07" containerID="94729f9ba718870d628d3304ed83f8301d4aa333bc6358aa6e7b7c8ab50bf877" exitCode=0
Sep 16 19:05:15.354680 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.354683    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"
Sep 16 19:05:15.354884 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.354727    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr" event={"ID":"475a82f9-a998-440b-b0c5-8885dc7f2f07","Type":"ContainerDied","Data":"94729f9ba718870d628d3304ed83f8301d4aa333bc6358aa6e7b7c8ab50bf877"}
Sep 16 19:05:15.354884 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.354771    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr" event={"ID":"475a82f9-a998-440b-b0c5-8885dc7f2f07","Type":"ContainerDied","Data":"790e32c0d73dad05d69188fa9fa5b8e316d02068c352dcf3df34fd0cdf0f3859"}
Sep 16 19:05:15.354884 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.354797    2561 scope.go:117] "RemoveContainer" containerID="94729f9ba718870d628d3304ed83f8301d4aa333bc6358aa6e7b7c8ab50bf877"
Sep 16 19:05:15.364129 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.364107    2561 scope.go:117] "RemoveContainer" containerID="94729f9ba718870d628d3304ed83f8301d4aa333bc6358aa6e7b7c8ab50bf877"
Sep 16 19:05:15.364402 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:05:15.364384    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94729f9ba718870d628d3304ed83f8301d4aa333bc6358aa6e7b7c8ab50bf877\": container with ID starting with 94729f9ba718870d628d3304ed83f8301d4aa333bc6358aa6e7b7c8ab50bf877 not found: ID does not exist" containerID="94729f9ba718870d628d3304ed83f8301d4aa333bc6358aa6e7b7c8ab50bf877"
Sep 16 19:05:15.364445 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.364412    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94729f9ba718870d628d3304ed83f8301d4aa333bc6358aa6e7b7c8ab50bf877"} err="failed to get container status \"94729f9ba718870d628d3304ed83f8301d4aa333bc6358aa6e7b7c8ab50bf877\": rpc error: code = NotFound desc = could not find container \"94729f9ba718870d628d3304ed83f8301d4aa333bc6358aa6e7b7c8ab50bf877\": container with ID starting with 94729f9ba718870d628d3304ed83f8301d4aa333bc6358aa6e7b7c8ab50bf877 not found: ID does not exist"
Sep 16 19:05:15.379770 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.379740    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"]
Sep 16 19:05:15.383970 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:15.383946    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96wpvr"]
Sep 16 19:05:16.735254 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:16.735220    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="475a82f9-a998-440b-b0c5-8885dc7f2f07" path="/var/lib/kubelet/pods/475a82f9-a998-440b-b0c5-8885dc7f2f07/volumes"
Sep 16 19:05:19.357156 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:19.357127    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:19.454083 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:19.454010    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:24.466553 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:24.466512    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:25.984164 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:25.984127    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:32.041503 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.041464    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"]
Sep 16 19:05:32.042049 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.042000    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="475a82f9-a998-440b-b0c5-8885dc7f2f07" containerName="main"
Sep 16 19:05:32.042049 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.042023    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="475a82f9-a998-440b-b0c5-8885dc7f2f07" containerName="main"
Sep 16 19:05:32.042213 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.042169    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="475a82f9-a998-440b-b0c5-8885dc7f2f07" containerName="main"
Sep 16 19:05:32.044740 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.044724    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.048809 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.048786    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 19:05:32.048936 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.048905    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 19:05:32.048936 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.048905    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-qfhxz\""
Sep 16 19:05:32.049027 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.048951    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 19:05:32.054815 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.054791    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"]
Sep 16 19:05:32.138239 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.138208    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:32.181853 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.181813    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.182050 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.181861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.182050 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.181892    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.182050 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.181948    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm2gm\" (UniqueName: \"kubernetes.io/projected/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-kube-api-access-sm2gm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.182050 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.182001    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.182050 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.182052    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.282630 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.282597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.282825 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.282658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm2gm\" (UniqueName: \"kubernetes.io/projected/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-kube-api-access-sm2gm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.282825 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.282687    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.282825 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.282713    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.282825 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.282769    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.282825 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.282810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.283165 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.283024    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.283165 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.283089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.283242 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.283162    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.286079 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.286036    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.286322 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.286298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.294676 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.294603    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm2gm\" (UniqueName: \"kubernetes.io/projected/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-kube-api-access-sm2gm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.309864 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.309829    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"]
Sep 16 19:05:32.314338 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.314312    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"
Sep 16 19:05:32.317457 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.317427    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-5lnfz\""
Sep 16 19:05:32.327290 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.327261    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"]
Sep 16 19:05:32.338286 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.338257    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:32.355510 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.355479    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:32.384406 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.384341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/192ea168-8035-4423-a84f-2e3f057fc2e7-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c\" (UID: \"192ea168-8035-4423-a84f-2e3f057fc2e7\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"
Sep 16 19:05:32.384620 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.384435    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbbk9\" (UniqueName: \"kubernetes.io/projected/192ea168-8035-4423-a84f-2e3f057fc2e7-kube-api-access-hbbk9\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c\" (UID: \"192ea168-8035-4423-a84f-2e3f057fc2e7\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"
Sep 16 19:05:32.441847 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.441752    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:32.485546 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.485506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/192ea168-8035-4423-a84f-2e3f057fc2e7-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c\" (UID: \"192ea168-8035-4423-a84f-2e3f057fc2e7\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"
Sep 16 19:05:32.485754 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.485557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbbk9\" (UniqueName: \"kubernetes.io/projected/192ea168-8035-4423-a84f-2e3f057fc2e7-kube-api-access-hbbk9\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c\" (UID: \"192ea168-8035-4423-a84f-2e3f057fc2e7\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"
Sep 16 19:05:32.488478 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.488451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/192ea168-8035-4423-a84f-2e3f057fc2e7-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c\" (UID: \"192ea168-8035-4423-a84f-2e3f057fc2e7\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"
Sep 16 19:05:32.502905 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.502876    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbbk9\" (UniqueName: \"kubernetes.io/projected/192ea168-8035-4423-a84f-2e3f057fc2e7-kube-api-access-hbbk9\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c\" (UID: \"192ea168-8035-4423-a84f-2e3f057fc2e7\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"
Sep 16 19:05:32.632468 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.632388    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"
Sep 16 19:05:32.652208 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.652182    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"]
Sep 16 19:05:32.654341 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:05:32.654312    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf2af07_7ccc_4aa7_b4fd_31d5d3812be6.slice/crio-ab8e23fc669beb5630f09e45b96573b76dda0710cdb70b50bcce0c0f003044ee WatchSource:0}: Error finding container ab8e23fc669beb5630f09e45b96573b76dda0710cdb70b50bcce0c0f003044ee: Status 404 returned error can't find the container with id ab8e23fc669beb5630f09e45b96573b76dda0710cdb70b50bcce0c0f003044ee
Sep 16 19:05:32.656785 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.656764    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:05:32.659457 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.659432    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:32.913574 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:32.913552    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"]
Sep 16 19:05:32.915222 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:05:32.915191    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192ea168_8035_4423_a84f_2e3f057fc2e7.slice/crio-9a34612e8610310364feb42664a1ef4fab2358545bd240f1edb5e8d206e15ff3 WatchSource:0}: Error finding container 9a34612e8610310364feb42664a1ef4fab2358545bd240f1edb5e8d206e15ff3: Status 404 returned error can't find the container with id 9a34612e8610310364feb42664a1ef4fab2358545bd240f1edb5e8d206e15ff3
Sep 16 19:05:33.461701 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:33.461631    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c" event={"ID":"192ea168-8035-4423-a84f-2e3f057fc2e7","Type":"ContainerStarted","Data":"0762fbb0e56f530b3037bd35d5a8e0a78a497dcffb9a2d97da200ef3f15a7cb0"}
Sep 16 19:05:33.461701 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:33.461698    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c" event={"ID":"192ea168-8035-4423-a84f-2e3f057fc2e7","Type":"ContainerStarted","Data":"9a34612e8610310364feb42664a1ef4fab2358545bd240f1edb5e8d206e15ff3"}
Sep 16 19:05:33.462552 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:33.462532    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"
Sep 16 19:05:33.464385 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:33.464347    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g" event={"ID":"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6","Type":"ContainerStarted","Data":"ab8e23fc669beb5630f09e45b96573b76dda0710cdb70b50bcce0c0f003044ee"}
Sep 16 19:05:33.486955 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:33.486893    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c" podStartSLOduration=1.486869341 podStartE2EDuration="1.486869341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:33.484290507 +0000 UTC m=+1233.287008725" watchObservedRunningTime="2026-09-16 19:05:33.486869341 +0000 UTC m=+1233.289587558"
Sep 16 19:05:34.472116 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:34.472047    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g" event={"ID":"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6","Type":"ContainerStarted","Data":"e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08"}
Sep 16 19:05:34.492101 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:34.492013    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g" podStartSLOduration=0.790926109 podStartE2EDuration="2.491997739s" totalImagesPullingTime="1.70107163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:34.490006525 +0000 UTC m=+1234.292724767" watchObservedRunningTime="2026-09-16 19:05:34.491997739 +0000 UTC m=+1234.294715973"
Sep 16 19:05:35.356285 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:35.356249    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:38.606218 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:38.606183    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:39.407826 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:39.407757    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:39.474405 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:39.474372    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:42.356467 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:42.356423    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:42.369207 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:42.369178    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:05:42.531878 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:05:42.531845    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:06:05.479871 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:05.479834    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c" podUID="192ea168-8035-4423-a84f-2e3f057fc2e7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:05.548425 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:05.548306    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"]
Sep 16 19:06:05.551904 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:05.551868    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"]
Sep 16 19:06:05.554110 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:05.554076    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c" podUID="192ea168-8035-4423-a84f-2e3f057fc2e7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:05.559473 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:06:05.559436    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:06:05.559473 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:06:05.559473 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:06:05.559473 ip-10-0-129-125 kubenswrapper[2561]:  > containerID="e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08" cmd=["/bin/sleep","15"]
Sep 16 19:06:05.559700 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:05.559494    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g" message="" err=<
Sep 16 19:06:05.559700 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:06:05.559700 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:06:05.559700 ip-10-0-129-125 kubenswrapper[2561]:  >
Sep 16 19:06:05.559700 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:06:05.559506    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:06:05.559700 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:06:05.559700 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:06:05.559700 ip-10-0-129-125 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g" podUID="1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6" containerName="main" containerID="cri-o://e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08"
Sep 16 19:06:05.559700 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:05.559541    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g" podUID="1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6" containerName="main" containerID="cri-o://e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08" gracePeriod=60
Sep 16 19:06:05.571678 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:05.571632    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:06.319343 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.319317    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:06:06.403172 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.403138    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-tls-certs\") pod \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") "
Sep 16 19:06:06.403361 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.403216    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-tmp-dir\") pod \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") "
Sep 16 19:06:06.403361 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.403236    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-dshm\") pod \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") "
Sep 16 19:06:06.403361 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.403261    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm2gm\" (UniqueName: \"kubernetes.io/projected/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-kube-api-access-sm2gm\") pod \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") "
Sep 16 19:06:06.403361 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.403282    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-home\") pod \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") "
Sep 16 19:06:06.403361 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.403318    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-model-cache\") pod \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\" (UID: \"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6\") "
Sep 16 19:06:06.403638 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.403538    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6" (UID: "1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:06.403638 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.403580    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-home" (OuterVolumeSpecName: "home") pod "1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6" (UID: "1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:06.403638 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.403592    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-model-cache" (OuterVolumeSpecName: "model-cache") pod "1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6" (UID: "1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:06.405719 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.405689    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-kube-api-access-sm2gm" (OuterVolumeSpecName: "kube-api-access-sm2gm") pod "1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6" (UID: "1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6"). InnerVolumeSpecName "kube-api-access-sm2gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:06.405719 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.405712    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6" (UID: "1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:06.405857 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.405710    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-dshm" (OuterVolumeSpecName: "dshm") pod "1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6" (UID: "1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:06.504313 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.504278    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm2gm\" (UniqueName: \"kubernetes.io/projected/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-kube-api-access-sm2gm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:06:06.504313 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.504309    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:06:06.504313 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.504320    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:06:06.504877 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.504329    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:06:06.504877 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.504360    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:06:06.504877 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.504369    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:06:06.658357 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.658314    2561 generic.go:356] "Generic (PLEG): container finished" podID="1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6" containerID="e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08" exitCode=0
Sep 16 19:06:06.658357 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.658360    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"
Sep 16 19:06:06.658636 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.658399    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g" event={"ID":"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6","Type":"ContainerDied","Data":"e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08"}
Sep 16 19:06:06.658636 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.658466    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g" event={"ID":"1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6","Type":"ContainerDied","Data":"ab8e23fc669beb5630f09e45b96573b76dda0710cdb70b50bcce0c0f003044ee"}
Sep 16 19:06:06.658636 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.658497    2561 scope.go:117] "RemoveContainer" containerID="e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08"
Sep 16 19:06:06.669805 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.669785    2561 scope.go:117] "RemoveContainer" containerID="e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08"
Sep 16 19:06:06.670137 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:06:06.670117    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08\": container with ID starting with e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08 not found: ID does not exist" containerID="e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08"
Sep 16 19:06:06.670214 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.670143    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08"} err="failed to get container status \"e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08\": rpc error: code = NotFound desc = could not find container \"e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08\": container with ID starting with e3e883b31f36e17b0428057698398e47c0e77d603d141df3c2b257d8a205cd08 not found: ID does not exist"
Sep 16 19:06:06.683409 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.683380    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"]
Sep 16 19:06:06.687888 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.687863    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qsp8g"]
Sep 16 19:06:06.735174 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.735139    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6" path="/var/lib/kubelet/pods/1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6/volumes"
Sep 16 19:06:06.742045 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:06.742015    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:15.553977 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:15.553932    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c" podUID="192ea168-8035-4423-a84f-2e3f057fc2e7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:20.573552 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:20.573494    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c" podUID="192ea168-8035-4423-a84f-2e3f057fc2e7" containerName="main" containerID="cri-o://0762fbb0e56f530b3037bd35d5a8e0a78a497dcffb9a2d97da200ef3f15a7cb0" gracePeriod=45
Sep 16 19:06:25.553691 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:25.553648    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c" podUID="192ea168-8035-4423-a84f-2e3f057fc2e7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:35.554576 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:35.554538    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c" podUID="192ea168-8035-4423-a84f-2e3f057fc2e7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:44.438008 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:44.437965    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:44.538514 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:44.538470    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:45.553752 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:45.553711    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c" podUID="192ea168-8035-4423-a84f-2e3f057fc2e7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:50.832360 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:50.832328    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"
Sep 16 19:06:50.905621 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:50.905536    2561 generic.go:356] "Generic (PLEG): container finished" podID="192ea168-8035-4423-a84f-2e3f057fc2e7" containerID="0762fbb0e56f530b3037bd35d5a8e0a78a497dcffb9a2d97da200ef3f15a7cb0" exitCode=0
Sep 16 19:06:50.905621 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:50.905585    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"
Sep 16 19:06:50.905818 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:50.905617    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c" event={"ID":"192ea168-8035-4423-a84f-2e3f057fc2e7","Type":"ContainerDied","Data":"0762fbb0e56f530b3037bd35d5a8e0a78a497dcffb9a2d97da200ef3f15a7cb0"}
Sep 16 19:06:50.905818 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:50.905670    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c" event={"ID":"192ea168-8035-4423-a84f-2e3f057fc2e7","Type":"ContainerDied","Data":"9a34612e8610310364feb42664a1ef4fab2358545bd240f1edb5e8d206e15ff3"}
Sep 16 19:06:50.905818 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:50.905694    2561 scope.go:117] "RemoveContainer" containerID="0762fbb0e56f530b3037bd35d5a8e0a78a497dcffb9a2d97da200ef3f15a7cb0"
Sep 16 19:06:50.914507 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:50.914485    2561 scope.go:117] "RemoveContainer" containerID="0762fbb0e56f530b3037bd35d5a8e0a78a497dcffb9a2d97da200ef3f15a7cb0"
Sep 16 19:06:50.914790 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:06:50.914770    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0762fbb0e56f530b3037bd35d5a8e0a78a497dcffb9a2d97da200ef3f15a7cb0\": container with ID starting with 0762fbb0e56f530b3037bd35d5a8e0a78a497dcffb9a2d97da200ef3f15a7cb0 not found: ID does not exist" containerID="0762fbb0e56f530b3037bd35d5a8e0a78a497dcffb9a2d97da200ef3f15a7cb0"
Sep 16 19:06:50.914837 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:50.914801    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0762fbb0e56f530b3037bd35d5a8e0a78a497dcffb9a2d97da200ef3f15a7cb0"} err="failed to get container status \"0762fbb0e56f530b3037bd35d5a8e0a78a497dcffb9a2d97da200ef3f15a7cb0\": rpc error: code = NotFound desc = could not find container \"0762fbb0e56f530b3037bd35d5a8e0a78a497dcffb9a2d97da200ef3f15a7cb0\": container with ID starting with 0762fbb0e56f530b3037bd35d5a8e0a78a497dcffb9a2d97da200ef3f15a7cb0 not found: ID does not exist"
Sep 16 19:06:50.921136 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:50.921113    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbbk9\" (UniqueName: \"kubernetes.io/projected/192ea168-8035-4423-a84f-2e3f057fc2e7-kube-api-access-hbbk9\") pod \"192ea168-8035-4423-a84f-2e3f057fc2e7\" (UID: \"192ea168-8035-4423-a84f-2e3f057fc2e7\") "
Sep 16 19:06:50.921251 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:50.921203    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/192ea168-8035-4423-a84f-2e3f057fc2e7-tls-certs\") pod \"192ea168-8035-4423-a84f-2e3f057fc2e7\" (UID: \"192ea168-8035-4423-a84f-2e3f057fc2e7\") "
Sep 16 19:06:50.923316 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:50.923281    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192ea168-8035-4423-a84f-2e3f057fc2e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "192ea168-8035-4423-a84f-2e3f057fc2e7" (UID: "192ea168-8035-4423-a84f-2e3f057fc2e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:50.923439 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:50.923395    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192ea168-8035-4423-a84f-2e3f057fc2e7-kube-api-access-hbbk9" (OuterVolumeSpecName: "kube-api-access-hbbk9") pod "192ea168-8035-4423-a84f-2e3f057fc2e7" (UID: "192ea168-8035-4423-a84f-2e3f057fc2e7"). InnerVolumeSpecName "kube-api-access-hbbk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:51.021762 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:51.021718    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/192ea168-8035-4423-a84f-2e3f057fc2e7-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:06:51.021762 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:51.021751    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbbk9\" (UniqueName: \"kubernetes.io/projected/192ea168-8035-4423-a84f-2e3f057fc2e7-kube-api-access-hbbk9\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:06:51.229284 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:51.229245    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"]
Sep 16 19:06:51.234548 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:51.234517    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b9n6c"]
Sep 16 19:06:52.734489 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:52.734446    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="192ea168-8035-4423-a84f-2e3f057fc2e7" path="/var/lib/kubelet/pods/192ea168-8035-4423-a84f-2e3f057fc2e7/volumes"
Sep 16 19:06:55.637156 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:55.637123    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:55.747270 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:06:55.747233    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:02.483619 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:02.483558    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:08.792513 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:08.792480    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:08.865127 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:08.865096    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:10.553855 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.553823    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"]
Sep 16 19:07:10.554379 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.554300    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6" containerName="main"
Sep 16 19:07:10.554379 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.554314    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6" containerName="main"
Sep 16 19:07:10.554379 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.554328    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="192ea168-8035-4423-a84f-2e3f057fc2e7" containerName="main"
Sep 16 19:07:10.554379 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.554334    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="192ea168-8035-4423-a84f-2e3f057fc2e7" containerName="main"
Sep 16 19:07:10.554607 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.554400    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="192ea168-8035-4423-a84f-2e3f057fc2e7" containerName="main"
Sep 16 19:07:10.554607 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.554407    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cf2af07-7ccc-4aa7-b4fd-31d5d3812be6" containerName="main"
Sep 16 19:07:10.558860 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.558845    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.562041 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.562020    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 16 19:07:10.563419 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.563397    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-2xzts\""
Sep 16 19:07:10.563553 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.563394    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 16 19:07:10.563553 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.563455    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 16 19:07:10.568377 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.568356    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"]
Sep 16 19:07:10.593838 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.593802    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c992f18-f442-48fa-ab64-1cde36d6a6ad-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.594021 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.593859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.594021 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.593949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.594021 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.593987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.594181 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.594018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2jqk\" (UniqueName: \"kubernetes.io/projected/6c992f18-f442-48fa-ab64-1cde36d6a6ad-kube-api-access-r2jqk\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.594181 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.594071    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.656586 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.656550    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:10.694797 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.694759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.694987 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.694812    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.694987 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.694838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.694987 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.694863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2jqk\" (UniqueName: \"kubernetes.io/projected/6c992f18-f442-48fa-ab64-1cde36d6a6ad-kube-api-access-r2jqk\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.694987 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.694882    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.694987 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.694911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c992f18-f442-48fa-ab64-1cde36d6a6ad-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.695338 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.695314    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.695409 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.695351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.695409 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.695381    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.697386 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.697362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.697507 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.697474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c992f18-f442-48fa-ab64-1cde36d6a6ad-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.704115 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.704083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2jqk\" (UniqueName: \"kubernetes.io/projected/6c992f18-f442-48fa-ab64-1cde36d6a6ad-kube-api-access-r2jqk\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-rxsd2\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.870621 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.870523    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:10.904671 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:10.904629    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:11.162071 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:11.162027    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"]
Sep 16 19:07:11.163319 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:07:11.163290    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c992f18_f442_48fa_ab64_1cde36d6a6ad.slice/crio-b0e36f5d0eca933cc8bde23fef4b7db1532d4c91b28097e6edbc1c1469567503 WatchSource:0}: Error finding container b0e36f5d0eca933cc8bde23fef4b7db1532d4c91b28097e6edbc1c1469567503: Status 404 returned error can't find the container with id b0e36f5d0eca933cc8bde23fef4b7db1532d4c91b28097e6edbc1c1469567503
Sep 16 19:07:12.031563 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:12.031524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2" event={"ID":"6c992f18-f442-48fa-ab64-1cde36d6a6ad","Type":"ContainerStarted","Data":"7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17"}
Sep 16 19:07:12.031978 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:12.031587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2" event={"ID":"6c992f18-f442-48fa-ab64-1cde36d6a6ad","Type":"ContainerStarted","Data":"b0e36f5d0eca933cc8bde23fef4b7db1532d4c91b28097e6edbc1c1469567503"}
Sep 16 19:07:12.058160 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:12.058103    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2" podStartSLOduration=2.058082674 podStartE2EDuration="2.058082674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:12.05549826 +0000 UTC m=+1331.858216481" watchObservedRunningTime="2026-09-16 19:07:12.058082674 +0000 UTC m=+1331.860800891"
Sep 16 19:07:12.870796 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:12.870744    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:17.452733 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:17.452700    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:20.871752 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:20.871710    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:20.884156 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:20.884127    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:21.094968 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:21.094883    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:22.213869 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:22.213834    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:22.286149 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:22.286120    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:45.134395 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.134358    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"]
Sep 16 19:07:45.144500 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:07:45.144457    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:07:45.144500 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:07:45.144500 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:07:45.144500 ip-10-0-129-125 kubenswrapper[2561]:  > containerID="7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17" cmd=["/bin/sleep","15"]
Sep 16 19:07:45.144755 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.144519    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2" message="" err=<
Sep 16 19:07:45.144755 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:07:45.144755 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:07:45.144755 ip-10-0-129-125 kubenswrapper[2561]:  >
Sep 16 19:07:45.144755 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:07:45.144531    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:07:45.144755 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:07:45.144755 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:07:45.144755 ip-10-0-129-125 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2" podUID="6c992f18-f442-48fa-ab64-1cde36d6a6ad" containerName="main" containerID="cri-o://7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17"
Sep 16 19:07:45.144755 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.144585    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2" podUID="6c992f18-f442-48fa-ab64-1cde36d6a6ad" containerName="main" containerID="cri-o://7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17" gracePeriod=60
Sep 16 19:07:45.148563 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.148502    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:45.218338 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.218306    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:45.901947 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.901923    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:45.938643 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.938613    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-model-cache\") pod \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") "
Sep 16 19:07:45.938838 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.938704    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c992f18-f442-48fa-ab64-1cde36d6a6ad-tls-certs\") pod \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") "
Sep 16 19:07:45.938838 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.938723    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-tmp-dir\") pod \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") "
Sep 16 19:07:45.938838 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.938742    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-dshm\") pod \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") "
Sep 16 19:07:45.938838 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.938761    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2jqk\" (UniqueName: \"kubernetes.io/projected/6c992f18-f442-48fa-ab64-1cde36d6a6ad-kube-api-access-r2jqk\") pod \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") "
Sep 16 19:07:45.938838 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.938813    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-home\") pod \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\" (UID: \"6c992f18-f442-48fa-ab64-1cde36d6a6ad\") "
Sep 16 19:07:45.939150 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.938906    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c992f18-f442-48fa-ab64-1cde36d6a6ad" (UID: "6c992f18-f442-48fa-ab64-1cde36d6a6ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:45.939150 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.939008    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c992f18-f442-48fa-ab64-1cde36d6a6ad" (UID: "6c992f18-f442-48fa-ab64-1cde36d6a6ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:45.939259 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.939217    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:07:45.939259 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.939241    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:07:45.939259 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.939248    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-home" (OuterVolumeSpecName: "home") pod "6c992f18-f442-48fa-ab64-1cde36d6a6ad" (UID: "6c992f18-f442-48fa-ab64-1cde36d6a6ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:45.941475 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.941442    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c992f18-f442-48fa-ab64-1cde36d6a6ad-kube-api-access-r2jqk" (OuterVolumeSpecName: "kube-api-access-r2jqk") pod "6c992f18-f442-48fa-ab64-1cde36d6a6ad" (UID: "6c992f18-f442-48fa-ab64-1cde36d6a6ad"). InnerVolumeSpecName "kube-api-access-r2jqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:45.941760 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.941736    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c992f18-f442-48fa-ab64-1cde36d6a6ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c992f18-f442-48fa-ab64-1cde36d6a6ad" (UID: "6c992f18-f442-48fa-ab64-1cde36d6a6ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:45.941760 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:45.941736    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-dshm" (OuterVolumeSpecName: "dshm") pod "6c992f18-f442-48fa-ab64-1cde36d6a6ad" (UID: "6c992f18-f442-48fa-ab64-1cde36d6a6ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:46.040726 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:46.040630    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c992f18-f442-48fa-ab64-1cde36d6a6ad-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:07:46.040726 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:46.040660    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:07:46.040726 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:46.040674    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2jqk\" (UniqueName: \"kubernetes.io/projected/6c992f18-f442-48fa-ab64-1cde36d6a6ad-kube-api-access-r2jqk\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:07:46.040726 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:46.040687    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c992f18-f442-48fa-ab64-1cde36d6a6ad-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:07:46.220834 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:46.220803    2561 generic.go:356] "Generic (PLEG): container finished" podID="6c992f18-f442-48fa-ab64-1cde36d6a6ad" containerID="7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17" exitCode=0
Sep 16 19:07:46.221298 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:46.220848    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"
Sep 16 19:07:46.221298 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:46.220876    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2" event={"ID":"6c992f18-f442-48fa-ab64-1cde36d6a6ad","Type":"ContainerDied","Data":"7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17"}
Sep 16 19:07:46.221298 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:46.220917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2" event={"ID":"6c992f18-f442-48fa-ab64-1cde36d6a6ad","Type":"ContainerDied","Data":"b0e36f5d0eca933cc8bde23fef4b7db1532d4c91b28097e6edbc1c1469567503"}
Sep 16 19:07:46.221298 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:46.220938    2561 scope.go:117] "RemoveContainer" containerID="7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17"
Sep 16 19:07:46.231359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:46.231342    2561 scope.go:117] "RemoveContainer" containerID="7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17"
Sep 16 19:07:46.231669 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:07:46.231639    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17\": container with ID starting with 7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17 not found: ID does not exist" containerID="7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17"
Sep 16 19:07:46.231710 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:46.231680    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17"} err="failed to get container status \"7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17\": rpc error: code = NotFound desc = could not find container \"7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17\": container with ID starting with 7b3bb117a2d1a83cb0bd1d365f87cc35db18d67896d4b0985ad9d91215a38b17 not found: ID does not exist"
Sep 16 19:07:46.245397 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:46.245368    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"]
Sep 16 19:07:46.251399 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:46.251370    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-rxsd2"]
Sep 16 19:07:46.735958 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:07:46.735922    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c992f18-f442-48fa-ab64-1cde36d6a6ad" path="/var/lib/kubelet/pods/6c992f18-f442-48fa-ab64-1cde36d6a6ad/volumes"
Sep 16 19:08:10.868372 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:10.868340    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:10.976882 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:10.976841    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:16.412474 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:16.412431    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:17.075620 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:17.075586    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:19.402032 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.401992    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:19.628941 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.628905    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"]
Sep 16 19:08:19.629475 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.629452    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c992f18-f442-48fa-ab64-1cde36d6a6ad" containerName="main"
Sep 16 19:08:19.629475 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.629474    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c992f18-f442-48fa-ab64-1cde36d6a6ad" containerName="main"
Sep 16 19:08:19.629626 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.629533    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c992f18-f442-48fa-ab64-1cde36d6a6ad" containerName="main"
Sep 16 19:08:19.633797 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.633775    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"
Sep 16 19:08:19.636635 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.636611    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 19:08:19.636786 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.636745    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 19:08:19.638082 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.638042    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 19:08:19.638186 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.638115    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-92jmj\""
Sep 16 19:08:19.638186 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.638124    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-scqd8\""
Sep 16 19:08:19.642765 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.642736    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"]
Sep 16 19:08:19.744182 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.744135    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:19.747435 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.747414    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3feb87-54e8-4f7b-9ba6-10772200fbf7-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl\" (UID: \"7e3feb87-54e8-4f7b-9ba6-10772200fbf7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"
Sep 16 19:08:19.747520 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.747478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvkfr\" (UniqueName: \"kubernetes.io/projected/7e3feb87-54e8-4f7b-9ba6-10772200fbf7-kube-api-access-wvkfr\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl\" (UID: \"7e3feb87-54e8-4f7b-9ba6-10772200fbf7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"
Sep 16 19:08:19.848820 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.848777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3feb87-54e8-4f7b-9ba6-10772200fbf7-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl\" (UID: \"7e3feb87-54e8-4f7b-9ba6-10772200fbf7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"
Sep 16 19:08:19.849044 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.848866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvkfr\" (UniqueName: \"kubernetes.io/projected/7e3feb87-54e8-4f7b-9ba6-10772200fbf7-kube-api-access-wvkfr\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl\" (UID: \"7e3feb87-54e8-4f7b-9ba6-10772200fbf7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"
Sep 16 19:08:19.851731 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.851709    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3feb87-54e8-4f7b-9ba6-10772200fbf7-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl\" (UID: \"7e3feb87-54e8-4f7b-9ba6-10772200fbf7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"
Sep 16 19:08:19.857571 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.857551    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvkfr\" (UniqueName: \"kubernetes.io/projected/7e3feb87-54e8-4f7b-9ba6-10772200fbf7-kube-api-access-wvkfr\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl\" (UID: \"7e3feb87-54e8-4f7b-9ba6-10772200fbf7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"
Sep 16 19:08:19.945500 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:19.945467    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"
Sep 16 19:08:20.222951 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:20.222911    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"]
Sep 16 19:08:20.225192 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:08:20.225145    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3feb87_54e8_4f7b_9ba6_10772200fbf7.slice/crio-09266767ad6384b7e0f6e96dbd92fc9efd2376bfe452353db4a30d5b33013792 WatchSource:0}: Error finding container 09266767ad6384b7e0f6e96dbd92fc9efd2376bfe452353db4a30d5b33013792: Status 404 returned error can't find the container with id 09266767ad6384b7e0f6e96dbd92fc9efd2376bfe452353db4a30d5b33013792
Sep 16 19:08:20.409226 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:20.409186    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl" event={"ID":"7e3feb87-54e8-4f7b-9ba6-10772200fbf7","Type":"ContainerStarted","Data":"e96f466021381b760ad032212d339c755c936100a73c2ad526bd819ca1d7f5b3"}
Sep 16 19:08:20.409711 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:20.409257    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"
Sep 16 19:08:20.409711 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:20.409283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl" event={"ID":"7e3feb87-54e8-4f7b-9ba6-10772200fbf7","Type":"ContainerStarted","Data":"09266767ad6384b7e0f6e96dbd92fc9efd2376bfe452353db4a30d5b33013792"}
Sep 16 19:08:20.428357 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:20.428302    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl" podStartSLOduration=1.428285923 podStartE2EDuration="1.428285923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:20.426752528 +0000 UTC m=+1400.229470749" watchObservedRunningTime="2026-09-16 19:08:20.428285923 +0000 UTC m=+1400.231004139"
Sep 16 19:08:23.599438 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:23.599404    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:23.698982 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:23.698946    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:32.478413 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:32.478366    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:33.970950 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:33.970921    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:34.063050 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:34.063014    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:38.572458 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:38.572422    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:38.905379 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:38.905296    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:39.262390 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:39.262356    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:41.007346 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:41.007314    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:42.248397 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:42.248361    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:44.422834 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:44.422793    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:48.445656 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:48.445618    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:48.529481 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:48.529446    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:51.416896 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:08:51.416816    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"
Sep 16 19:09:19.410595 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:09:19.410557    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:19.479571 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:09:19.479533    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:00.802737 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:10:00.802709    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:10:00.804495 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:10:00.804474    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:10:09.500159 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:10:09.500122    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:09.606640 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:10:09.606604    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:15.869600 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:10:15.869562    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:22.029396 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:10:22.029300    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:22.122294 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:10:22.122255    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:38.520391 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:10:38.520349    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:38.580561 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:10:38.580520    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:51.012867 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:10:51.012829    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:57.740499 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:10:57.740463    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:57.753776 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:10:57.753745    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"]
Sep 16 19:10:57.839288 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:10:57.839252    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:01.417003 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:01.416966    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl" podUID="7e3feb87-54e8-4f7b-9ba6-10772200fbf7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:11.416439 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:11.416396    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl" podUID="7e3feb87-54e8-4f7b-9ba6-10772200fbf7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:11.845592 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:11.845556    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:12.774189 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:12.774116    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl" podUID="7e3feb87-54e8-4f7b-9ba6-10772200fbf7" containerName="main" containerID="cri-o://e96f466021381b760ad032212d339c755c936100a73c2ad526bd819ca1d7f5b3" gracePeriod=45
Sep 16 19:11:13.112556 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:13.112469    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:21.416736 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:21.416699    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl" podUID="7e3feb87-54e8-4f7b-9ba6-10772200fbf7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:21.417167 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:21.416810    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"
Sep 16 19:11:31.416680 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:31.416639    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl" podUID="7e3feb87-54e8-4f7b-9ba6-10772200fbf7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:41.417076 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:41.417026    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl" podUID="7e3feb87-54e8-4f7b-9ba6-10772200fbf7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:43.026006 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.025980    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"
Sep 16 19:11:43.113120 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.113010    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3feb87-54e8-4f7b-9ba6-10772200fbf7-tls-certs\") pod \"7e3feb87-54e8-4f7b-9ba6-10772200fbf7\" (UID: \"7e3feb87-54e8-4f7b-9ba6-10772200fbf7\") "
Sep 16 19:11:43.113320 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.113141    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvkfr\" (UniqueName: \"kubernetes.io/projected/7e3feb87-54e8-4f7b-9ba6-10772200fbf7-kube-api-access-wvkfr\") pod \"7e3feb87-54e8-4f7b-9ba6-10772200fbf7\" (UID: \"7e3feb87-54e8-4f7b-9ba6-10772200fbf7\") "
Sep 16 19:11:43.115447 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.115413    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e3feb87-54e8-4f7b-9ba6-10772200fbf7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e3feb87-54e8-4f7b-9ba6-10772200fbf7" (UID: "7e3feb87-54e8-4f7b-9ba6-10772200fbf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:43.115583 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.115482    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3feb87-54e8-4f7b-9ba6-10772200fbf7-kube-api-access-wvkfr" (OuterVolumeSpecName: "kube-api-access-wvkfr") pod "7e3feb87-54e8-4f7b-9ba6-10772200fbf7" (UID: "7e3feb87-54e8-4f7b-9ba6-10772200fbf7"). InnerVolumeSpecName "kube-api-access-wvkfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:43.214173 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.214126    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvkfr\" (UniqueName: \"kubernetes.io/projected/7e3feb87-54e8-4f7b-9ba6-10772200fbf7-kube-api-access-wvkfr\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:11:43.214173 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.214160    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3feb87-54e8-4f7b-9ba6-10772200fbf7-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:11:43.572394 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.572363    2561 generic.go:356] "Generic (PLEG): container finished" podID="7e3feb87-54e8-4f7b-9ba6-10772200fbf7" containerID="e96f466021381b760ad032212d339c755c936100a73c2ad526bd819ca1d7f5b3" exitCode=0
Sep 16 19:11:43.572584 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.572410    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"
Sep 16 19:11:43.572584 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.572447    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl" event={"ID":"7e3feb87-54e8-4f7b-9ba6-10772200fbf7","Type":"ContainerDied","Data":"e96f466021381b760ad032212d339c755c936100a73c2ad526bd819ca1d7f5b3"}
Sep 16 19:11:43.572584 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.572495    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl" event={"ID":"7e3feb87-54e8-4f7b-9ba6-10772200fbf7","Type":"ContainerDied","Data":"09266767ad6384b7e0f6e96dbd92fc9efd2376bfe452353db4a30d5b33013792"}
Sep 16 19:11:43.572584 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.572517    2561 scope.go:117] "RemoveContainer" containerID="e96f466021381b760ad032212d339c755c936100a73c2ad526bd819ca1d7f5b3"
Sep 16 19:11:43.582629 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.582611    2561 scope.go:117] "RemoveContainer" containerID="e96f466021381b760ad032212d339c755c936100a73c2ad526bd819ca1d7f5b3"
Sep 16 19:11:43.582984 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:11:43.582959    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e96f466021381b760ad032212d339c755c936100a73c2ad526bd819ca1d7f5b3\": container with ID starting with e96f466021381b760ad032212d339c755c936100a73c2ad526bd819ca1d7f5b3 not found: ID does not exist" containerID="e96f466021381b760ad032212d339c755c936100a73c2ad526bd819ca1d7f5b3"
Sep 16 19:11:43.583029 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.582995    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96f466021381b760ad032212d339c755c936100a73c2ad526bd819ca1d7f5b3"} err="failed to get container status \"e96f466021381b760ad032212d339c755c936100a73c2ad526bd819ca1d7f5b3\": rpc error: code = NotFound desc = could not find container \"e96f466021381b760ad032212d339c755c936100a73c2ad526bd819ca1d7f5b3\": container with ID starting with e96f466021381b760ad032212d339c755c936100a73c2ad526bd819ca1d7f5b3 not found: ID does not exist"
Sep 16 19:11:43.597699 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.597669    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"]
Sep 16 19:11:43.605617 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:43.605588    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-c2tsl"]
Sep 16 19:11:44.734527 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:44.734493    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e3feb87-54e8-4f7b-9ba6-10772200fbf7" path="/var/lib/kubelet/pods/7e3feb87-54e8-4f7b-9ba6-10772200fbf7/volumes"
Sep 16 19:11:47.842422 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:47.842381    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:47.942699 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:47.942664    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:48.056016 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:48.055985    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:52.268556 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.268474    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"]
Sep 16 19:11:52.268989 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.268914    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3feb87-54e8-4f7b-9ba6-10772200fbf7" containerName="main"
Sep 16 19:11:52.268989 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.268934    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3feb87-54e8-4f7b-9ba6-10772200fbf7" containerName="main"
Sep 16 19:11:52.269154 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.269132    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3feb87-54e8-4f7b-9ba6-10772200fbf7" containerName="main"
Sep 16 19:11:52.274116 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.274093    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.280956 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.280929    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-dwlcf\""
Sep 16 19:11:52.280956 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.280954    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 16 19:11:52.281192 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.280929    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 16 19:11:52.281192 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.280929    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 16 19:11:52.286836 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.286809    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"]
Sep 16 19:11:52.313660 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.313622    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:52.396965 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.396920    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.397167 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.397099    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-model-cache\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.397167 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.397151    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-dshm\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.397282 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.397260    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.397322 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.397304    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntpbn\" (UniqueName: \"kubernetes.io/projected/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-kube-api-access-ntpbn\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.397359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.397332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-home\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.498388 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.498349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-model-cache\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.498388 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.498399    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-dshm\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.498631 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.498529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.498631 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.498565    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntpbn\" (UniqueName: \"kubernetes.io/projected/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-kube-api-access-ntpbn\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.498631 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.498592    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-home\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.498761 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.498638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.498844 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.498819    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-model-cache\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.498963 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.498941    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.499027 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.498964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-home\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.500964 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.500942    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-dshm\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.501295 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.501276    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.510830 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.510808    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntpbn\" (UniqueName: \"kubernetes.io/projected/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-kube-api-access-ntpbn\") pod \"canary-v1-kserve-65d58ccc9-4gtwl\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.586443 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.586355    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:52.892539 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.892508    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"]
Sep 16 19:11:52.893524 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:11:52.893494    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd2e6b6_db2f_4fbd_b7fe_763a72097947.slice/crio-029879f7d4b60bbac88b065b63c6aeb05380ba11845c7d6c2b1bbbdcc5af0638 WatchSource:0}: Error finding container 029879f7d4b60bbac88b065b63c6aeb05380ba11845c7d6c2b1bbbdcc5af0638: Status 404 returned error can't find the container with id 029879f7d4b60bbac88b065b63c6aeb05380ba11845c7d6c2b1bbbdcc5af0638
Sep 16 19:11:52.895904 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:52.895883    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:11:53.642675 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:53.642632    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl" event={"ID":"2fd2e6b6-db2f-4fbd-b7fe-763a72097947","Type":"ContainerStarted","Data":"029879f7d4b60bbac88b065b63c6aeb05380ba11845c7d6c2b1bbbdcc5af0638"}
Sep 16 19:11:54.650203 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:54.650161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl" event={"ID":"2fd2e6b6-db2f-4fbd-b7fe-763a72097947","Type":"ContainerStarted","Data":"2499db5e991ccc4d06c1aa835c0eec38d5240399f047205c98cc4a459b87b8f3"}
Sep 16 19:11:54.677858 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:54.677778    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl" podStartSLOduration=1.653449196 podStartE2EDuration="2.6777596s" totalImagesPullingTime="1.024310404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:54.677239267 +0000 UTC m=+1614.479957484" watchObservedRunningTime="2026-09-16 19:11:54.6777596 +0000 UTC m=+1614.480477816"
Sep 16 19:11:55.587109 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:55.587073    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:11:55.934899 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:11:55.934796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:01.143551 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:01.143517    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:01.897399 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:01.897362    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:02.012939 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:02.012900    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:02.587053 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:02.587014    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:12:02.600568 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:02.600535    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:12:02.707772 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:02.707745    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:12:04.469379 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:04.469340    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:04.572312 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:04.572279    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:08.961527 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:08.961486    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:16.074311 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:16.074250    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:16.241561 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:16.241527    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:20.060923 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:20.060892    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:28.139627 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.139585    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"]
Sep 16 19:12:28.145986 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.145958    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.150220 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.150196    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 19:12:28.150868 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.150851    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:12:28.151514 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.151463    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 19:12:28.151514 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.151481    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-c2vgg\""
Sep 16 19:12:28.151679 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.151486    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 19:12:28.151679 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.151487    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 19:12:28.172287 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.172254    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"]
Sep 16 19:12:28.198734 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.198697    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:28.230645 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.230603    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxzp2\" (UniqueName: \"kubernetes.io/projected/fdc1c8e3-1195-4f98-816e-a79400c4af8d-kube-api-access-cxzp2\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.230645 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.230642    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc1c8e3-1195-4f98-816e-a79400c4af8d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.230906 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.230666    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-model-cache\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.230906 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.230781    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-dshm\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.230906 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.230826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-home\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.230906 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.230850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.230906 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.230896    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdc1c8e3-1195-4f98-816e-a79400c4af8d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.231180 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.230967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.332026 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.331978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-dshm\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.332026 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.332030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-home\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.332312 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.332079    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.332312 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.332124    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdc1c8e3-1195-4f98-816e-a79400c4af8d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.332312 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.332176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.332312 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.332254    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxzp2\" (UniqueName: \"kubernetes.io/projected/fdc1c8e3-1195-4f98-816e-a79400c4af8d-kube-api-access-cxzp2\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.332312 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.332285    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc1c8e3-1195-4f98-816e-a79400c4af8d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.332577 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.332327    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-model-cache\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.332577 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.332519    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-home\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.332694 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.332614    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.332694 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.332661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.332805 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.332703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-model-cache\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.332864 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.332847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdc1c8e3-1195-4f98-816e-a79400c4af8d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.334458 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.334437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-dshm\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.335070 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.335029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc1c8e3-1195-4f98-816e-a79400c4af8d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.342122 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.342048    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxzp2\" (UniqueName: \"kubernetes.io/projected/fdc1c8e3-1195-4f98-816e-a79400c4af8d-kube-api-access-cxzp2\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-4nvgp\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.457572 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.457469    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:28.642041 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.642000    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:28.728359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.728324    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:28.747488 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.747459    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"]
Sep 16 19:12:28.749761 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:12:28.749732    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc1c8e3_1195_4f98_816e_a79400c4af8d.slice/crio-4135e987985d874ec842cd574b75e1a4612f1931017eae1870cb2eba33a021d0 WatchSource:0}: Error finding container 4135e987985d874ec842cd574b75e1a4612f1931017eae1870cb2eba33a021d0: Status 404 returned error can't find the container with id 4135e987985d874ec842cd574b75e1a4612f1931017eae1870cb2eba33a021d0
Sep 16 19:12:28.841539 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.841499    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp" event={"ID":"fdc1c8e3-1195-4f98-816e-a79400c4af8d","Type":"ContainerStarted","Data":"4ff3e51f9a50ba244d875778318afdb9e1ba1e3a9285f05ef9118a7b5d499d74"}
Sep 16 19:12:28.841714 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:28.841561    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp" event={"ID":"fdc1c8e3-1195-4f98-816e-a79400c4af8d","Type":"ContainerStarted","Data":"4135e987985d874ec842cd574b75e1a4612f1931017eae1870cb2eba33a021d0"}
Sep 16 19:12:31.860397 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:31.860356    2561 generic.go:356] "Generic (PLEG): container finished" podID="fdc1c8e3-1195-4f98-816e-a79400c4af8d" containerID="4ff3e51f9a50ba244d875778318afdb9e1ba1e3a9285f05ef9118a7b5d499d74" exitCode=0
Sep 16 19:12:31.860786 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:31.860432    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp" event={"ID":"fdc1c8e3-1195-4f98-816e-a79400c4af8d","Type":"ContainerDied","Data":"4ff3e51f9a50ba244d875778318afdb9e1ba1e3a9285f05ef9118a7b5d499d74"}
Sep 16 19:12:32.462950 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:32.462902    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:32.868912 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:32.868822    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp" event={"ID":"fdc1c8e3-1195-4f98-816e-a79400c4af8d","Type":"ContainerStarted","Data":"a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8"}
Sep 16 19:12:33.458394 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:33.458353    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:38.457733 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:38.457684    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:38.470535 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:38.470507    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:38.498761 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:38.498695    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp" podStartSLOduration=8.023092948 podStartE2EDuration="10.498678108s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.47558516s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:32.933028402 +0000 UTC m=+1652.735746619" watchObservedRunningTime="2026-09-16 19:12:38.498678108 +0000 UTC m=+1658.301396361"
Sep 16 19:12:38.916335 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:38.916305    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:12:40.344014 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:40.343941    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:40.791408 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:40.791373    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:51.489453 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:12:51.489418    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:40.866863 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:13:40.866831    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:40.966396 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:13:40.966347    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:53.144780 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:13:53.144736    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:53.255461 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:13:53.255421    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:53.420915 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:13:53.420823    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:54.278115 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:13:54.278048    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"]
Sep 16 19:13:54.286029 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:13:54.285999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:55.176599 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:13:55.176560    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:58.206660 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:13:58.206614    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:06.469163 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.469127    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"]
Sep 16 19:14:06.472749 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.472723    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.475417 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.475388    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 16 19:14:06.475543 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.475416    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-lmqzv\""
Sep 16 19:14:06.475543 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.475415    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 19:14:06.475543 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.475463    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 16 19:14:06.490926 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.490898    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"]
Sep 16 19:14:06.506929 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.506893    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:06.555102 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.555042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-dshm\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.555311 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.555117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9z8s\" (UniqueName: \"kubernetes.io/projected/c3ce447f-a37b-4c8e-b8db-8661f749722b-kube-api-access-g9z8s\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.555311 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.555155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-model-cache\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.555311 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.555195    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.555500 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.555335    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ce447f-a37b-4c8e-b8db-8661f749722b-tls-certs\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.555500 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.555391    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-home\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.656129 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.656088    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-model-cache\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.656313 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.656146    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.656313 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.656204    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ce447f-a37b-4c8e-b8db-8661f749722b-tls-certs\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.656313 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.656240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-home\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.656313 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.656301    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-dshm\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.656482 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.656334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9z8s\" (UniqueName: \"kubernetes.io/projected/c3ce447f-a37b-4c8e-b8db-8661f749722b-kube-api-access-g9z8s\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.656580 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.656557    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-model-cache\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.656625 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.656593    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.656690 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.656670    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-home\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.658729 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.658708    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-dshm\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.658980 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.658962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ce447f-a37b-4c8e-b8db-8661f749722b-tls-certs\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.671097 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.671048    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9z8s\" (UniqueName: \"kubernetes.io/projected/c3ce447f-a37b-4c8e-b8db-8661f749722b-kube-api-access-g9z8s\") pod \"test-div-v1-kserve-fb566d667-lmqbd\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:06.784868 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:06.784770    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:07.068670 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:07.068641    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"]
Sep 16 19:14:07.070774 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:14:07.070741    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ce447f_a37b_4c8e_b8db_8661f749722b.slice/crio-166ef4f6374f3fa205bfde50add195fbb6d1249baaaf1ec063e39bd7e91a4d09 WatchSource:0}: Error finding container 166ef4f6374f3fa205bfde50add195fbb6d1249baaaf1ec063e39bd7e91a4d09: Status 404 returned error can't find the container with id 166ef4f6374f3fa205bfde50add195fbb6d1249baaaf1ec063e39bd7e91a4d09
Sep 16 19:14:07.410847 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:07.410766    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd" event={"ID":"c3ce447f-a37b-4c8e-b8db-8661f749722b","Type":"ContainerStarted","Data":"4126b18bc3d0472030f1796f58b91956f9b56bf44b95ff331896379a927d76bb"}
Sep 16 19:14:07.410847 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:07.410824    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd" event={"ID":"c3ce447f-a37b-4c8e-b8db-8661f749722b","Type":"ContainerStarted","Data":"166ef4f6374f3fa205bfde50add195fbb6d1249baaaf1ec063e39bd7e91a4d09"}
Sep 16 19:14:07.443043 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:07.442980    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd" podStartSLOduration=1.442960655 podStartE2EDuration="1.442960655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:07.441040738 +0000 UTC m=+1747.243758954" watchObservedRunningTime="2026-09-16 19:14:07.442960655 +0000 UTC m=+1747.245678904"
Sep 16 19:14:07.785521 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:07.785485    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:09.307312 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.307271    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl" podUID="2fd2e6b6-db2f-4fbd-b7fe-763a72097947" containerName="main" containerID="cri-o://2499db5e991ccc4d06c1aa835c0eec38d5240399f047205c98cc4a459b87b8f3" gracePeriod=45
Sep 16 19:14:09.426882 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.426848    2561 generic.go:356] "Generic (PLEG): container finished" podID="2fd2e6b6-db2f-4fbd-b7fe-763a72097947" containerID="2499db5e991ccc4d06c1aa835c0eec38d5240399f047205c98cc4a459b87b8f3" exitCode=0
Sep 16 19:14:09.427030 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.426918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl" event={"ID":"2fd2e6b6-db2f-4fbd-b7fe-763a72097947","Type":"ContainerDied","Data":"2499db5e991ccc4d06c1aa835c0eec38d5240399f047205c98cc4a459b87b8f3"}
Sep 16 19:14:09.558484 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.558409    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:14:09.686016 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.685979    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-model-cache\") pod \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") "
Sep 16 19:14:09.686284 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.686036    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-dshm\") pod \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") "
Sep 16 19:14:09.686284 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.686088    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-home\") pod \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") "
Sep 16 19:14:09.686284 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.686121    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-tls-certs\") pod \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") "
Sep 16 19:14:09.686284 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.686137    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntpbn\" (UniqueName: \"kubernetes.io/projected/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-kube-api-access-ntpbn\") pod \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") "
Sep 16 19:14:09.686284 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.686187    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-tmp-dir\") pod \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\" (UID: \"2fd2e6b6-db2f-4fbd-b7fe-763a72097947\") "
Sep 16 19:14:09.686579 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.686367    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fd2e6b6-db2f-4fbd-b7fe-763a72097947" (UID: "2fd2e6b6-db2f-4fbd-b7fe-763a72097947"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:09.686579 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.686380    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-home" (OuterVolumeSpecName: "home") pod "2fd2e6b6-db2f-4fbd-b7fe-763a72097947" (UID: "2fd2e6b6-db2f-4fbd-b7fe-763a72097947"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:09.686579 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.686560    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:14:09.686579 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.686573    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:14:09.686766 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.686584    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fd2e6b6-db2f-4fbd-b7fe-763a72097947" (UID: "2fd2e6b6-db2f-4fbd-b7fe-763a72097947"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:09.688743 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.688717    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-dshm" (OuterVolumeSpecName: "dshm") pod "2fd2e6b6-db2f-4fbd-b7fe-763a72097947" (UID: "2fd2e6b6-db2f-4fbd-b7fe-763a72097947"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:09.688743 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.688723    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fd2e6b6-db2f-4fbd-b7fe-763a72097947" (UID: "2fd2e6b6-db2f-4fbd-b7fe-763a72097947"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:09.688885 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.688743    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-kube-api-access-ntpbn" (OuterVolumeSpecName: "kube-api-access-ntpbn") pod "2fd2e6b6-db2f-4fbd-b7fe-763a72097947" (UID: "2fd2e6b6-db2f-4fbd-b7fe-763a72097947"). InnerVolumeSpecName "kube-api-access-ntpbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:09.787394 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.787348    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:14:09.787394 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.787393    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:14:09.787677 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.787406    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:14:09.787677 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:09.787419    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntpbn\" (UniqueName: \"kubernetes.io/projected/2fd2e6b6-db2f-4fbd-b7fe-763a72097947-kube-api-access-ntpbn\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:14:10.434103 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:10.434035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl" event={"ID":"2fd2e6b6-db2f-4fbd-b7fe-763a72097947","Type":"ContainerDied","Data":"029879f7d4b60bbac88b065b63c6aeb05380ba11845c7d6c2b1bbbdcc5af0638"}
Sep 16 19:14:10.434625 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:10.434112    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"
Sep 16 19:14:10.434625 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:10.434174    2561 scope.go:117] "RemoveContainer" containerID="2499db5e991ccc4d06c1aa835c0eec38d5240399f047205c98cc4a459b87b8f3"
Sep 16 19:14:10.466743 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:10.466712    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"]
Sep 16 19:14:10.471734 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:10.471709    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4gtwl"]
Sep 16 19:14:10.735384 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:10.735352    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fd2e6b6-db2f-4fbd-b7fe-763a72097947" path="/var/lib/kubelet/pods/2fd2e6b6-db2f-4fbd-b7fe-763a72097947/volumes"
Sep 16 19:14:16.784957 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:16.784914    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:16.798186 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:16.798160    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:17.486886 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:17.486857    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:14:20.168122 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.168088    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"]
Sep 16 19:14:20.168722 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.168696    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fd2e6b6-db2f-4fbd-b7fe-763a72097947" containerName="main"
Sep 16 19:14:20.168722 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.168722    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fd2e6b6-db2f-4fbd-b7fe-763a72097947" containerName="main"
Sep 16 19:14:20.168897 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.168804    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fd2e6b6-db2f-4fbd-b7fe-763a72097947" containerName="main"
Sep 16 19:14:20.173255 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.173238    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.178584 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.178553    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 19:14:20.191784 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.191747    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"]
Sep 16 19:14:20.216635 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.216596    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:20.289070 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.289023    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.289293 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.289092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.289293 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.289118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.289293 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.289133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6214dd-86a0-4a57-949f-a55bef4b7859-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.289293 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.289229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-home\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.289496 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.289355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqz97\" (UniqueName: \"kubernetes.io/projected/9b6214dd-86a0-4a57-949f-a55bef4b7859-kube-api-access-dqz97\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.390477 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.390441    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-home\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.390685 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.390511    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqz97\" (UniqueName: \"kubernetes.io/projected/9b6214dd-86a0-4a57-949f-a55bef4b7859-kube-api-access-dqz97\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.390685 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.390535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.390685 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.390552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.390685 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.390574    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.390685 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.390590    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6214dd-86a0-4a57-949f-a55bef4b7859-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.390948 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.390914    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-home\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.390948 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.390933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.391093 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.391002    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.393284 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.393255    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.393399 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.393354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6214dd-86a0-4a57-949f-a55bef4b7859-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.400890 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.400848    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqz97\" (UniqueName: \"kubernetes.io/projected/9b6214dd-86a0-4a57-949f-a55bef4b7859-kube-api-access-dqz97\") pod \"test-div-v2-kserve-5f6fdb6b77-wlwt2\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.484038 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.484004    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:20.767444 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:20.767415    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"]
Sep 16 19:14:21.508136 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:21.508102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2" event={"ID":"9b6214dd-86a0-4a57-949f-a55bef4b7859","Type":"ContainerStarted","Data":"9cf010dc02dc61db01d65f087e0990b752cb2de51e0afc99a584305cc5a0c0ca"}
Sep 16 19:14:21.508616 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:21.508178    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2" event={"ID":"9b6214dd-86a0-4a57-949f-a55bef4b7859","Type":"ContainerStarted","Data":"bded40ae4bc4ebcefaa39da8322a8eebf050c3211d4243c3eb3aa600e43622fa"}
Sep 16 19:14:21.536048 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:21.535990    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2" podStartSLOduration=1.535973778 podStartE2EDuration="1.535973778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:21.53411968 +0000 UTC m=+1761.336837897" watchObservedRunningTime="2026-09-16 19:14:21.535973778 +0000 UTC m=+1761.338691994"
Sep 16 19:14:22.485102 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:22.485067    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:30.484532 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:30.484493    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:30.498160 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:30.498124    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:30.569944 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:30.569909    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:14:35.586339 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:35.586283    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:41.655006 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:41.654972    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:41.755036 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:41.755002    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:45.819098 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:45.819032    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:45.834398 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:45.834368    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"]
Sep 16 19:14:45.845961 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:14:45.845918    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:14:45.845961 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:14:45.845961 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:14:45.845961 ip-10-0-129-125 kubenswrapper[2561]:  > containerID="a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8" cmd=["/bin/sleep","15"]
Sep 16 19:14:45.846312 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:45.845984    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp" message="" err=<
Sep 16 19:14:45.846312 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:14:45.846312 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:14:45.846312 ip-10-0-129-125 kubenswrapper[2561]:  >
Sep 16 19:14:45.846312 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:14:45.846001    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:14:45.846312 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:14:45.846312 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:14:45.846312 ip-10-0-129-125 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp" podUID="fdc1c8e3-1195-4f98-816e-a79400c4af8d" containerName="main" containerID="cri-o://a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8"
Sep 16 19:14:45.846312 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:45.846088    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp" podUID="fdc1c8e3-1195-4f98-816e-a79400c4af8d" containerName="main" containerID="cri-o://a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8" gracePeriod=60
Sep 16 19:14:45.905677 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:45.905625    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp" podUID="fdc1c8e3-1195-4f98-816e-a79400c4af8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:14:45.918072 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:45.918029    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:46.622073 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.622032    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:14:46.656624 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.656583    2561 generic.go:356] "Generic (PLEG): container finished" podID="fdc1c8e3-1195-4f98-816e-a79400c4af8d" containerID="a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8" exitCode=0
Sep 16 19:14:46.656788 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.656636    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"
Sep 16 19:14:46.656788 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.656656    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp" event={"ID":"fdc1c8e3-1195-4f98-816e-a79400c4af8d","Type":"ContainerDied","Data":"a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8"}
Sep 16 19:14:46.656788 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.656718    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp" event={"ID":"fdc1c8e3-1195-4f98-816e-a79400c4af8d","Type":"ContainerDied","Data":"4135e987985d874ec842cd574b75e1a4612f1931017eae1870cb2eba33a021d0"}
Sep 16 19:14:46.656788 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.656750    2561 scope.go:117] "RemoveContainer" containerID="a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8"
Sep 16 19:14:46.666483 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.666463    2561 scope.go:117] "RemoveContainer" containerID="4ff3e51f9a50ba244d875778318afdb9e1ba1e3a9285f05ef9118a7b5d499d74"
Sep 16 19:14:46.675189 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.675168    2561 scope.go:117] "RemoveContainer" containerID="a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8"
Sep 16 19:14:46.675473 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:14:46.675451    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8\": container with ID starting with a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8 not found: ID does not exist" containerID="a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8"
Sep 16 19:14:46.675560 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.675486    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8"} err="failed to get container status \"a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8\": rpc error: code = NotFound desc = could not find container \"a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8\": container with ID starting with a52555240e19da6e7439fdeaad42e67562092e53752e0c2704095adc16bb69c8 not found: ID does not exist"
Sep 16 19:14:46.675560 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.675514    2561 scope.go:117] "RemoveContainer" containerID="4ff3e51f9a50ba244d875778318afdb9e1ba1e3a9285f05ef9118a7b5d499d74"
Sep 16 19:14:46.675833 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:14:46.675814    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff3e51f9a50ba244d875778318afdb9e1ba1e3a9285f05ef9118a7b5d499d74\": container with ID starting with 4ff3e51f9a50ba244d875778318afdb9e1ba1e3a9285f05ef9118a7b5d499d74 not found: ID does not exist" containerID="4ff3e51f9a50ba244d875778318afdb9e1ba1e3a9285f05ef9118a7b5d499d74"
Sep 16 19:14:46.675882 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.675840    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff3e51f9a50ba244d875778318afdb9e1ba1e3a9285f05ef9118a7b5d499d74"} err="failed to get container status \"4ff3e51f9a50ba244d875778318afdb9e1ba1e3a9285f05ef9118a7b5d499d74\": rpc error: code = NotFound desc = could not find container \"4ff3e51f9a50ba244d875778318afdb9e1ba1e3a9285f05ef9118a7b5d499d74\": container with ID starting with 4ff3e51f9a50ba244d875778318afdb9e1ba1e3a9285f05ef9118a7b5d499d74 not found: ID does not exist"
Sep 16 19:14:46.735311 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.735279    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc1c8e3-1195-4f98-816e-a79400c4af8d-tls-certs\") pod \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") "
Sep 16 19:14:46.735516 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.735355    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-tmp-dir\") pod \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") "
Sep 16 19:14:46.735516 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.735387    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdc1c8e3-1195-4f98-816e-a79400c4af8d-cabundle-cert\") pod \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") "
Sep 16 19:14:46.735516 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.735450    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-home\") pod \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") "
Sep 16 19:14:46.735516 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.735509    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-kserve-provision-location\") pod \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") "
Sep 16 19:14:46.735762 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.735563    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-dshm\") pod \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") "
Sep 16 19:14:46.735762 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.735594    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxzp2\" (UniqueName: \"kubernetes.io/projected/fdc1c8e3-1195-4f98-816e-a79400c4af8d-kube-api-access-cxzp2\") pod \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") "
Sep 16 19:14:46.735762 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.735618    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-model-cache\") pod \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\" (UID: \"fdc1c8e3-1195-4f98-816e-a79400c4af8d\") "
Sep 16 19:14:46.735762 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.735612    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdc1c8e3-1195-4f98-816e-a79400c4af8d" (UID: "fdc1c8e3-1195-4f98-816e-a79400c4af8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:46.735984 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.735801    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-home" (OuterVolumeSpecName: "home") pod "fdc1c8e3-1195-4f98-816e-a79400c4af8d" (UID: "fdc1c8e3-1195-4f98-816e-a79400c4af8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:46.735984 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.735819    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc1c8e3-1195-4f98-816e-a79400c4af8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fdc1c8e3-1195-4f98-816e-a79400c4af8d" (UID: "fdc1c8e3-1195-4f98-816e-a79400c4af8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:46.736127 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.736017    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdc1c8e3-1195-4f98-816e-a79400c4af8d" (UID: "fdc1c8e3-1195-4f98-816e-a79400c4af8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:46.736127 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.735942    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.736127 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.736106    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdc1c8e3-1195-4f98-816e-a79400c4af8d-cabundle-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.736127 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.736122    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.738043 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.738015    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-dshm" (OuterVolumeSpecName: "dshm") pod "fdc1c8e3-1195-4f98-816e-a79400c4af8d" (UID: "fdc1c8e3-1195-4f98-816e-a79400c4af8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:46.738211 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.738123    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc1c8e3-1195-4f98-816e-a79400c4af8d-kube-api-access-cxzp2" (OuterVolumeSpecName: "kube-api-access-cxzp2") pod "fdc1c8e3-1195-4f98-816e-a79400c4af8d" (UID: "fdc1c8e3-1195-4f98-816e-a79400c4af8d"). InnerVolumeSpecName "kube-api-access-cxzp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:46.738274 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.738215    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc1c8e3-1195-4f98-816e-a79400c4af8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdc1c8e3-1195-4f98-816e-a79400c4af8d" (UID: "fdc1c8e3-1195-4f98-816e-a79400c4af8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:46.758323 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.758264    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fdc1c8e3-1195-4f98-816e-a79400c4af8d" (UID: "fdc1c8e3-1195-4f98-816e-a79400c4af8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:46.837262 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.837220    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-kserve-provision-location\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.837262 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.837255    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.837262 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.837271    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxzp2\" (UniqueName: \"kubernetes.io/projected/fdc1c8e3-1195-4f98-816e-a79400c4af8d-kube-api-access-cxzp2\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.837760 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.837286    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc1c8e3-1195-4f98-816e-a79400c4af8d-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.837760 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.837302    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc1c8e3-1195-4f98-816e-a79400c4af8d-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.982694 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.982656    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"]
Sep 16 19:14:46.985673 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:46.985606    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-4nvgp"]
Sep 16 19:14:48.734759 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:48.734718    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdc1c8e3-1195-4f98-816e-a79400c4af8d" path="/var/lib/kubelet/pods/fdc1c8e3-1195-4f98-816e-a79400c4af8d/volumes"
Sep 16 19:14:49.644880 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:49.644834    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:49.666998 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:49.666964    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"]
Sep 16 19:14:49.745342 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:49.745204    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:49.850608 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:49.850577    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:49.968949 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:49.968910    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"]
Sep 16 19:14:50.001900 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:50.001869    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:54.710942 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:14:54.710906    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:00.474499 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:00.474464    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:00.853919 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:00.853838    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:15:00.859208 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:00.859181    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:15:04.688734 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:04.688686    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd" podUID="c3ce447f-a37b-4c8e-b8db-8661f749722b" containerName="main" containerID="cri-o://4126b18bc3d0472030f1796f58b91956f9b56bf44b95ff331896379a927d76bb" gracePeriod=45
Sep 16 19:15:04.951810 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:04.951747    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:15:04.990874 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:04.990821    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2" podUID="9b6214dd-86a0-4a57-949f-a55bef4b7859" containerName="main" containerID="cri-o://9cf010dc02dc61db01d65f087e0990b752cb2de51e0afc99a584305cc5a0c0ca" gracePeriod=45
Sep 16 19:15:05.119249 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.119212    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ce447f-a37b-4c8e-b8db-8661f749722b-tls-certs\") pod \"c3ce447f-a37b-4c8e-b8db-8661f749722b\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") "
Sep 16 19:15:05.119424 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.119279    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-tmp-dir\") pod \"c3ce447f-a37b-4c8e-b8db-8661f749722b\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") "
Sep 16 19:15:05.119424 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.119311    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-model-cache\") pod \"c3ce447f-a37b-4c8e-b8db-8661f749722b\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") "
Sep 16 19:15:05.119424 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.119341    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9z8s\" (UniqueName: \"kubernetes.io/projected/c3ce447f-a37b-4c8e-b8db-8661f749722b-kube-api-access-g9z8s\") pod \"c3ce447f-a37b-4c8e-b8db-8661f749722b\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") "
Sep 16 19:15:05.119424 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.119415    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-dshm\") pod \"c3ce447f-a37b-4c8e-b8db-8661f749722b\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") "
Sep 16 19:15:05.119656 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.119447    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-home\") pod \"c3ce447f-a37b-4c8e-b8db-8661f749722b\" (UID: \"c3ce447f-a37b-4c8e-b8db-8661f749722b\") "
Sep 16 19:15:05.119656 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.119590    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3ce447f-a37b-4c8e-b8db-8661f749722b" (UID: "c3ce447f-a37b-4c8e-b8db-8661f749722b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:05.119656 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.119605    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3ce447f-a37b-4c8e-b8db-8661f749722b" (UID: "c3ce447f-a37b-4c8e-b8db-8661f749722b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:05.119819 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.119802    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-home" (OuterVolumeSpecName: "home") pod "c3ce447f-a37b-4c8e-b8db-8661f749722b" (UID: "c3ce447f-a37b-4c8e-b8db-8661f749722b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:05.119878 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.119857    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:15:05.119935 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.119875    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:15:05.121904 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.121865    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ce447f-a37b-4c8e-b8db-8661f749722b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3ce447f-a37b-4c8e-b8db-8661f749722b" (UID: "c3ce447f-a37b-4c8e-b8db-8661f749722b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:05.121904 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.121894    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ce447f-a37b-4c8e-b8db-8661f749722b-kube-api-access-g9z8s" (OuterVolumeSpecName: "kube-api-access-g9z8s") pod "c3ce447f-a37b-4c8e-b8db-8661f749722b" (UID: "c3ce447f-a37b-4c8e-b8db-8661f749722b"). InnerVolumeSpecName "kube-api-access-g9z8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:05.122090 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.121932    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-dshm" (OuterVolumeSpecName: "dshm") pod "c3ce447f-a37b-4c8e-b8db-8661f749722b" (UID: "c3ce447f-a37b-4c8e-b8db-8661f749722b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:05.220827 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.220787    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:15:05.220827 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.220828    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ce447f-a37b-4c8e-b8db-8661f749722b-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:15:05.221070 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.220842    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ce447f-a37b-4c8e-b8db-8661f749722b-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:15:05.221070 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.220858    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9z8s\" (UniqueName: \"kubernetes.io/projected/c3ce447f-a37b-4c8e-b8db-8661f749722b-kube-api-access-g9z8s\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:15:05.246735 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.246708    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:15:05.423116 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.423083    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-model-cache\") pod \"9b6214dd-86a0-4a57-949f-a55bef4b7859\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") "
Sep 16 19:15:05.423324 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.423155    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-home\") pod \"9b6214dd-86a0-4a57-949f-a55bef4b7859\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") "
Sep 16 19:15:05.423324 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.423218    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6214dd-86a0-4a57-949f-a55bef4b7859-tls-certs\") pod \"9b6214dd-86a0-4a57-949f-a55bef4b7859\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") "
Sep 16 19:15:05.423324 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.423281    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-tmp-dir\") pod \"9b6214dd-86a0-4a57-949f-a55bef4b7859\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") "
Sep 16 19:15:05.423324 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.423308    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqz97\" (UniqueName: \"kubernetes.io/projected/9b6214dd-86a0-4a57-949f-a55bef4b7859-kube-api-access-dqz97\") pod \"9b6214dd-86a0-4a57-949f-a55bef4b7859\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") "
Sep 16 19:15:05.423527 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.423357    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-dshm\") pod \"9b6214dd-86a0-4a57-949f-a55bef4b7859\" (UID: \"9b6214dd-86a0-4a57-949f-a55bef4b7859\") "
Sep 16 19:15:05.423527 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.423499    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b6214dd-86a0-4a57-949f-a55bef4b7859" (UID: "9b6214dd-86a0-4a57-949f-a55bef4b7859"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:05.423622 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.423527    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-home" (OuterVolumeSpecName: "home") pod "9b6214dd-86a0-4a57-949f-a55bef4b7859" (UID: "9b6214dd-86a0-4a57-949f-a55bef4b7859"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:05.423622 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.423580    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b6214dd-86a0-4a57-949f-a55bef4b7859" (UID: "9b6214dd-86a0-4a57-949f-a55bef4b7859"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:05.423738 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.423720    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:15:05.423785 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.423746    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:15:05.423785 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.423761    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:15:05.425508 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.425477    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b6214dd-86a0-4a57-949f-a55bef4b7859-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b6214dd-86a0-4a57-949f-a55bef4b7859" (UID: "9b6214dd-86a0-4a57-949f-a55bef4b7859"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:05.425643 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.425619    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6214dd-86a0-4a57-949f-a55bef4b7859-kube-api-access-dqz97" (OuterVolumeSpecName: "kube-api-access-dqz97") pod "9b6214dd-86a0-4a57-949f-a55bef4b7859" (UID: "9b6214dd-86a0-4a57-949f-a55bef4b7859"). InnerVolumeSpecName "kube-api-access-dqz97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:05.425793 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.425772    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-dshm" (OuterVolumeSpecName: "dshm") pod "9b6214dd-86a0-4a57-949f-a55bef4b7859" (UID: "9b6214dd-86a0-4a57-949f-a55bef4b7859"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:05.525095 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.524972    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6214dd-86a0-4a57-949f-a55bef4b7859-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:15:05.525095 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.525008    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqz97\" (UniqueName: \"kubernetes.io/projected/9b6214dd-86a0-4a57-949f-a55bef4b7859-kube-api-access-dqz97\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:15:05.525095 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.525023    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6214dd-86a0-4a57-949f-a55bef4b7859-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:15:05.768856 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.768819    2561 generic.go:356] "Generic (PLEG): container finished" podID="9b6214dd-86a0-4a57-949f-a55bef4b7859" containerID="9cf010dc02dc61db01d65f087e0990b752cb2de51e0afc99a584305cc5a0c0ca" exitCode=0
Sep 16 19:15:05.769349 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.768862    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"
Sep 16 19:15:05.769349 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.768910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2" event={"ID":"9b6214dd-86a0-4a57-949f-a55bef4b7859","Type":"ContainerDied","Data":"9cf010dc02dc61db01d65f087e0990b752cb2de51e0afc99a584305cc5a0c0ca"}
Sep 16 19:15:05.769349 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.768970    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2" event={"ID":"9b6214dd-86a0-4a57-949f-a55bef4b7859","Type":"ContainerDied","Data":"bded40ae4bc4ebcefaa39da8322a8eebf050c3211d4243c3eb3aa600e43622fa"}
Sep 16 19:15:05.769349 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.769003    2561 scope.go:117] "RemoveContainer" containerID="9cf010dc02dc61db01d65f087e0990b752cb2de51e0afc99a584305cc5a0c0ca"
Sep 16 19:15:05.772336 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.771792    2561 generic.go:356] "Generic (PLEG): container finished" podID="c3ce447f-a37b-4c8e-b8db-8661f749722b" containerID="4126b18bc3d0472030f1796f58b91956f9b56bf44b95ff331896379a927d76bb" exitCode=0
Sep 16 19:15:05.772336 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.771828    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd" event={"ID":"c3ce447f-a37b-4c8e-b8db-8661f749722b","Type":"ContainerDied","Data":"4126b18bc3d0472030f1796f58b91956f9b56bf44b95ff331896379a927d76bb"}
Sep 16 19:15:05.772336 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.771867    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd" event={"ID":"c3ce447f-a37b-4c8e-b8db-8661f749722b","Type":"ContainerDied","Data":"166ef4f6374f3fa205bfde50add195fbb6d1249baaaf1ec063e39bd7e91a4d09"}
Sep 16 19:15:05.772336 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.771927    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"
Sep 16 19:15:05.782606 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.782578    2561 scope.go:117] "RemoveContainer" containerID="9cf010dc02dc61db01d65f087e0990b752cb2de51e0afc99a584305cc5a0c0ca"
Sep 16 19:15:05.782948 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:15:05.782924    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf010dc02dc61db01d65f087e0990b752cb2de51e0afc99a584305cc5a0c0ca\": container with ID starting with 9cf010dc02dc61db01d65f087e0990b752cb2de51e0afc99a584305cc5a0c0ca not found: ID does not exist" containerID="9cf010dc02dc61db01d65f087e0990b752cb2de51e0afc99a584305cc5a0c0ca"
Sep 16 19:15:05.783021 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.782957    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf010dc02dc61db01d65f087e0990b752cb2de51e0afc99a584305cc5a0c0ca"} err="failed to get container status \"9cf010dc02dc61db01d65f087e0990b752cb2de51e0afc99a584305cc5a0c0ca\": rpc error: code = NotFound desc = could not find container \"9cf010dc02dc61db01d65f087e0990b752cb2de51e0afc99a584305cc5a0c0ca\": container with ID starting with 9cf010dc02dc61db01d65f087e0990b752cb2de51e0afc99a584305cc5a0c0ca not found: ID does not exist"
Sep 16 19:15:05.783021 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.782975    2561 scope.go:117] "RemoveContainer" containerID="4126b18bc3d0472030f1796f58b91956f9b56bf44b95ff331896379a927d76bb"
Sep 16 19:15:05.791636 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.791620    2561 scope.go:117] "RemoveContainer" containerID="4126b18bc3d0472030f1796f58b91956f9b56bf44b95ff331896379a927d76bb"
Sep 16 19:15:05.791924 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:15:05.791903    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4126b18bc3d0472030f1796f58b91956f9b56bf44b95ff331896379a927d76bb\": container with ID starting with 4126b18bc3d0472030f1796f58b91956f9b56bf44b95ff331896379a927d76bb not found: ID does not exist" containerID="4126b18bc3d0472030f1796f58b91956f9b56bf44b95ff331896379a927d76bb"
Sep 16 19:15:05.791978 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.791930    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4126b18bc3d0472030f1796f58b91956f9b56bf44b95ff331896379a927d76bb"} err="failed to get container status \"4126b18bc3d0472030f1796f58b91956f9b56bf44b95ff331896379a927d76bb\": rpc error: code = NotFound desc = could not find container \"4126b18bc3d0472030f1796f58b91956f9b56bf44b95ff331896379a927d76bb\": container with ID starting with 4126b18bc3d0472030f1796f58b91956f9b56bf44b95ff331896379a927d76bb not found: ID does not exist"
Sep 16 19:15:05.800406 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.800381    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"]
Sep 16 19:15:05.805155 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.805124    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-lmqbd"]
Sep 16 19:15:05.817253 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.817219    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"]
Sep 16 19:15:05.821150 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:05.821124    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-wlwt2"]
Sep 16 19:15:06.734794 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:06.734760    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b6214dd-86a0-4a57-949f-a55bef4b7859" path="/var/lib/kubelet/pods/9b6214dd-86a0-4a57-949f-a55bef4b7859/volumes"
Sep 16 19:15:06.735186 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:06.735171    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3ce447f-a37b-4c8e-b8db-8661f749722b" path="/var/lib/kubelet/pods/c3ce447f-a37b-4c8e-b8db-8661f749722b/volumes"
Sep 16 19:15:10.539554 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.539519    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"]
Sep 16 19:15:10.539999 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.539917    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc1c8e3-1195-4f98-816e-a79400c4af8d" containerName="storage-initializer"
Sep 16 19:15:10.539999 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.539931    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc1c8e3-1195-4f98-816e-a79400c4af8d" containerName="storage-initializer"
Sep 16 19:15:10.539999 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.539946    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc1c8e3-1195-4f98-816e-a79400c4af8d" containerName="main"
Sep 16 19:15:10.539999 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.539951    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc1c8e3-1195-4f98-816e-a79400c4af8d" containerName="main"
Sep 16 19:15:10.539999 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.539958    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b6214dd-86a0-4a57-949f-a55bef4b7859" containerName="main"
Sep 16 19:15:10.539999 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.539963    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b6214dd-86a0-4a57-949f-a55bef4b7859" containerName="main"
Sep 16 19:15:10.539999 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.539976    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ce447f-a37b-4c8e-b8db-8661f749722b" containerName="main"
Sep 16 19:15:10.539999 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.539981    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ce447f-a37b-4c8e-b8db-8661f749722b" containerName="main"
Sep 16 19:15:10.540281 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.540035    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc1c8e3-1195-4f98-816e-a79400c4af8d" containerName="main"
Sep 16 19:15:10.540281 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.540043    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b6214dd-86a0-4a57-949f-a55bef4b7859" containerName="main"
Sep 16 19:15:10.540281 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.540051    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ce447f-a37b-4c8e-b8db-8661f749722b" containerName="main"
Sep 16 19:15:10.544889 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.544867    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.548927 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.548897    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-wc99b\""
Sep 16 19:15:10.549091 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.548898    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 16 19:15:10.549091 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.548905    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 19:15:10.549091 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.549083    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 16 19:15:10.553428 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.553402    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"]
Sep 16 19:15:10.653463 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.653404    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:10.677680 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.677645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.677680 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.677686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.677899 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.677749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nk9n\" (UniqueName: \"kubernetes.io/projected/34645dd9-5685-4259-a728-e6af0c4a992b-kube-api-access-4nk9n\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.677899 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.677810    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-home\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.677899 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.677837    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.677899 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.677867    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34645dd9-5685-4259-a728-e6af0c4a992b-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.779189 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.779147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nk9n\" (UniqueName: \"kubernetes.io/projected/34645dd9-5685-4259-a728-e6af0c4a992b-kube-api-access-4nk9n\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.779430 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.779232    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-home\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.779430 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.779263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.779430 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.779301    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34645dd9-5685-4259-a728-e6af0c4a992b-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.779430 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.779347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.779430 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.779380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.779766 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.779739    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-home\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.779864 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.779818    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.780188 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.780162    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.782553 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.782527    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.782693 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.782673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34645dd9-5685-4259-a728-e6af0c4a992b-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.788296 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.788272    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nk9n\" (UniqueName: \"kubernetes.io/projected/34645dd9-5685-4259-a728-e6af0c4a992b-kube-api-access-4nk9n\") pod \"test-div-v2-kserve-5f6fdb6b77-mnlps\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:10.856536 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:10.856433    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:11.138929 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:11.138893    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"]
Sep 16 19:15:11.141281 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:15:11.141242    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34645dd9_5685_4259_a728_e6af0c4a992b.slice/crio-55688ca12a52910fc3ab3de28ba53cd4fd2efaaff7619e17f8918269a9eae89f WatchSource:0}: Error finding container 55688ca12a52910fc3ab3de28ba53cd4fd2efaaff7619e17f8918269a9eae89f: Status 404 returned error can't find the container with id 55688ca12a52910fc3ab3de28ba53cd4fd2efaaff7619e17f8918269a9eae89f
Sep 16 19:15:11.810948 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:11.810905    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps" event={"ID":"34645dd9-5685-4259-a728-e6af0c4a992b","Type":"ContainerStarted","Data":"27492e7dea75a6a43b85c261f14ca5fd75f61895bc19ec55194f8971082e351b"}
Sep 16 19:15:11.811417 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:11.810969    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps" event={"ID":"34645dd9-5685-4259-a728-e6af0c4a992b","Type":"ContainerStarted","Data":"55688ca12a52910fc3ab3de28ba53cd4fd2efaaff7619e17f8918269a9eae89f"}
Sep 16 19:15:11.832862 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:11.832807    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps" podStartSLOduration=1.83279013 podStartE2EDuration="1.83279013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:11.830670025 +0000 UTC m=+1811.633388244" watchObservedRunningTime="2026-09-16 19:15:11.83279013 +0000 UTC m=+1811.635508363"
Sep 16 19:15:11.857081 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:11.857021    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:20.856970 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:20.856933    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:20.872529 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:20.872504    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:21.868824 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:21.868794    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:15:32.482492 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:32.482450    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:33.967939 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:33.967885    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:34.060928 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:34.060886    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:38.357498 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:38.357459    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:38.457411 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:38.457367    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:40.582318 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:40.582272    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:46.896989 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:46.896947    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:46.993592 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:15:46.993553    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:02.266945 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.266908    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"]
Sep 16 19:16:02.270930 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.270903    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.273874 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.273840    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 19:16:02.273874 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.273872    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 19:16:02.274111 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.273848    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:16:02.274752 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.274733    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 19:16:02.274850 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.274812    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 19:16:02.274926 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.274909    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-hqbzm\""
Sep 16 19:16:02.285009 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.284980    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"]
Sep 16 19:16:02.296136 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.296094    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:02.354307 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.354262    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-model-cache\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.354307 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.354307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.354527 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.354388    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72095731-e56a-418b-8ed4-1270704bbc6b-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.354527 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.354436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.354527 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.354511    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr9gg\" (UniqueName: \"kubernetes.io/projected/72095731-e56a-418b-8ed4-1270704bbc6b-kube-api-access-kr9gg\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.354636 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.354536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-dshm\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.354636 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.354581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72095731-e56a-418b-8ed4-1270704bbc6b-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.354636 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.354616    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-home\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.455409 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.455361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-home\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.455594 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.455432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-model-cache\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.455594 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.455458    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.455594 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.455485    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72095731-e56a-418b-8ed4-1270704bbc6b-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.455594 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.455514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.455594 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.455581    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr9gg\" (UniqueName: \"kubernetes.io/projected/72095731-e56a-418b-8ed4-1270704bbc6b-kube-api-access-kr9gg\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.455868 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.455607    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-dshm\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.455868 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.455649    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72095731-e56a-418b-8ed4-1270704bbc6b-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.455868 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.455833    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-home\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.456029 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.455937    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-model-cache\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.456121 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.456029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.456121 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.456093    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.456299 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.456281    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72095731-e56a-418b-8ed4-1270704bbc6b-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.458239 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.458221    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-dshm\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.458463 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.458446    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72095731-e56a-418b-8ed4-1270704bbc6b-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.464367 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.464346    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr9gg\" (UniqueName: \"kubernetes.io/projected/72095731-e56a-418b-8ed4-1270704bbc6b-kube-api-access-kr9gg\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.583904 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.583821    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:02.614999 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.614943    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:02.686221 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.686177    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:02.876907 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:02.876871    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"]
Sep 16 19:16:02.877912 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:16:02.877885    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72095731_e56a_418b_8ed4_1270704bbc6b.slice/crio-daf1795702e73c08eba3068fd5a50e1101e38a24280240b8b8029636ce64516b WatchSource:0}: Error finding container daf1795702e73c08eba3068fd5a50e1101e38a24280240b8b8029636ce64516b: Status 404 returned error can't find the container with id daf1795702e73c08eba3068fd5a50e1101e38a24280240b8b8029636ce64516b
Sep 16 19:16:03.108704 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:03.108663    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm" event={"ID":"72095731-e56a-418b-8ed4-1270704bbc6b","Type":"ContainerStarted","Data":"cd680bfa9ce263fb19232acd0c69eeaea2e9e1527d35e3e0dcb21c886597b1a1"}
Sep 16 19:16:03.108859 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:03.108722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm" event={"ID":"72095731-e56a-418b-8ed4-1270704bbc6b","Type":"ContainerStarted","Data":"daf1795702e73c08eba3068fd5a50e1101e38a24280240b8b8029636ce64516b"}
Sep 16 19:16:06.132044 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:06.132007    2561 generic.go:356] "Generic (PLEG): container finished" podID="72095731-e56a-418b-8ed4-1270704bbc6b" containerID="cd680bfa9ce263fb19232acd0c69eeaea2e9e1527d35e3e0dcb21c886597b1a1" exitCode=0
Sep 16 19:16:06.132503 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:06.132098    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm" event={"ID":"72095731-e56a-418b-8ed4-1270704bbc6b","Type":"ContainerDied","Data":"cd680bfa9ce263fb19232acd0c69eeaea2e9e1527d35e3e0dcb21c886597b1a1"}
Sep 16 19:16:07.139675 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:07.139633    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm" event={"ID":"72095731-e56a-418b-8ed4-1270704bbc6b","Type":"ContainerStarted","Data":"3cdf7980c3cd058e503bf8d78933af78cd8278008d6c1326cc5bc60ec5db7d0a"}
Sep 16 19:16:07.165293 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:07.165241    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm" podStartSLOduration=2.6580255619999997 podStartE2EDuration="5.16522515s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.507199588s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:07.162834503 +0000 UTC m=+1866.965552743" watchObservedRunningTime="2026-09-16 19:16:07.16522515 +0000 UTC m=+1866.967943416"
Sep 16 19:16:07.584561 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:07.584525    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:12.584273 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:12.584230    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:12.597032 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:12.597004    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:13.190469 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:13.190439    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:16:14.079468 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:14.079429    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:22.972189 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:22.972107    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:23.198834 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:23.198796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:23.494658 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:23.494620    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:23.674230 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:23.674194    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"]
Sep 16 19:16:23.674230 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:23.674209    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:23.791941 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:23.791856    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:24.507870 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:24.507833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:27.985974 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:27.985932    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:30.060455 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:30.060419    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:31.156158 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.156128    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"]
Sep 16 19:16:31.159492 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.159474    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.163311 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.163282    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-bq7kv\""
Sep 16 19:16:31.163464 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.163388    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 16 19:16:31.164298 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.164267    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 16 19:16:31.164433 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.164366    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 16 19:16:31.174441 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.174416    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"]
Sep 16 19:16:31.208437 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.208368    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:31.242510 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.242475    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-home\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.242703 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.242519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-dshm\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.242703 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.242588    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-model-cache\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.242703 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.242691    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kqrw\" (UniqueName: \"kubernetes.io/projected/4d2527d2-9191-4b1c-8ac3-49908ee1889c-kube-api-access-6kqrw\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.242873 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.242735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2527d2-9191-4b1c-8ac3-49908ee1889c-tls-certs\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.242873 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.242789    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.344010 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.343972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2527d2-9191-4b1c-8ac3-49908ee1889c-tls-certs\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.344248 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.344022    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.344248 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.344081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-home\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.344248 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.344107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-dshm\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.344248 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.344155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-model-cache\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.344248 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.344216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kqrw\" (UniqueName: \"kubernetes.io/projected/4d2527d2-9191-4b1c-8ac3-49908ee1889c-kube-api-access-6kqrw\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.344530 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.344491    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.344598 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.344535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-home\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.344598 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.344548    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-model-cache\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.346682 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.346660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-dshm\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.346984 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.346962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2527d2-9191-4b1c-8ac3-49908ee1889c-tls-certs\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.353402 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.353382    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kqrw\" (UniqueName: \"kubernetes.io/projected/4d2527d2-9191-4b1c-8ac3-49908ee1889c-kube-api-access-6kqrw\") pod \"stop-v1-kserve-5657b88c95-dw5gh\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.470241 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.470209    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:31.757901 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:31.757874    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"]
Sep 16 19:16:31.760381 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:16:31.760353    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2527d2_9191_4b1c_8ac3_49908ee1889c.slice/crio-ef69f7ddac57906c5ee70507fd7a4a2c62ee0833e3df9222315b0852d5fb6430 WatchSource:0}: Error finding container ef69f7ddac57906c5ee70507fd7a4a2c62ee0833e3df9222315b0852d5fb6430: Status 404 returned error can't find the container with id ef69f7ddac57906c5ee70507fd7a4a2c62ee0833e3df9222315b0852d5fb6430
Sep 16 19:16:32.290413 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:32.290371    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh" event={"ID":"4d2527d2-9191-4b1c-8ac3-49908ee1889c","Type":"ContainerStarted","Data":"2e27c7074297bb6bdd777dc083f62eb718e5a6669c8f27980462cd9ef4f90d1e"}
Sep 16 19:16:32.290966 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:32.290433    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh" event={"ID":"4d2527d2-9191-4b1c-8ac3-49908ee1889c","Type":"ContainerStarted","Data":"ef69f7ddac57906c5ee70507fd7a4a2c62ee0833e3df9222315b0852d5fb6430"}
Sep 16 19:16:32.313636 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:32.313579    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh" podStartSLOduration=1.3135630489999999 podStartE2EDuration="1.313563049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:32.310751301 +0000 UTC m=+1892.113469517" watchObservedRunningTime="2026-09-16 19:16:32.313563049 +0000 UTC m=+1892.116281269"
Sep 16 19:16:32.470371 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:32.470338    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:35.890025 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:35.889989    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:36.003424 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:36.003391    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:38.693814 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:38.693769    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps" podUID="34645dd9-5685-4259-a728-e6af0c4a992b" containerName="main" containerID="cri-o://27492e7dea75a6a43b85c261f14ca5fd75f61895bc19ec55194f8971082e351b" gracePeriod=45
Sep 16 19:16:38.857831 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:38.857784    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps" podUID="34645dd9-5685-4259-a728-e6af0c4a992b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:16:38.960955 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:38.960928    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:16:39.022494 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.022458    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-home\") pod \"34645dd9-5685-4259-a728-e6af0c4a992b\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") "
Sep 16 19:16:39.022662 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.022502    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34645dd9-5685-4259-a728-e6af0c4a992b-tls-certs\") pod \"34645dd9-5685-4259-a728-e6af0c4a992b\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") "
Sep 16 19:16:39.022662 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.022563    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-tmp-dir\") pod \"34645dd9-5685-4259-a728-e6af0c4a992b\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") "
Sep 16 19:16:39.022662 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.022608    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-dshm\") pod \"34645dd9-5685-4259-a728-e6af0c4a992b\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") "
Sep 16 19:16:39.022662 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.022638    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nk9n\" (UniqueName: \"kubernetes.io/projected/34645dd9-5685-4259-a728-e6af0c4a992b-kube-api-access-4nk9n\") pod \"34645dd9-5685-4259-a728-e6af0c4a992b\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") "
Sep 16 19:16:39.022826 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.022669    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-model-cache\") pod \"34645dd9-5685-4259-a728-e6af0c4a992b\" (UID: \"34645dd9-5685-4259-a728-e6af0c4a992b\") "
Sep 16 19:16:39.022826 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.022739    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-home" (OuterVolumeSpecName: "home") pod "34645dd9-5685-4259-a728-e6af0c4a992b" (UID: "34645dd9-5685-4259-a728-e6af0c4a992b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:39.022956 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.022930    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34645dd9-5685-4259-a728-e6af0c4a992b" (UID: "34645dd9-5685-4259-a728-e6af0c4a992b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:39.023004 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.022969    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-model-cache" (OuterVolumeSpecName: "model-cache") pod "34645dd9-5685-4259-a728-e6af0c4a992b" (UID: "34645dd9-5685-4259-a728-e6af0c4a992b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:39.023050 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.023010    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:16:39.025070 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.025031    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34645dd9-5685-4259-a728-e6af0c4a992b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34645dd9-5685-4259-a728-e6af0c4a992b" (UID: "34645dd9-5685-4259-a728-e6af0c4a992b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:39.025070 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.025036    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-dshm" (OuterVolumeSpecName: "dshm") pod "34645dd9-5685-4259-a728-e6af0c4a992b" (UID: "34645dd9-5685-4259-a728-e6af0c4a992b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:39.025253 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.025099    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34645dd9-5685-4259-a728-e6af0c4a992b-kube-api-access-4nk9n" (OuterVolumeSpecName: "kube-api-access-4nk9n") pod "34645dd9-5685-4259-a728-e6af0c4a992b" (UID: "34645dd9-5685-4259-a728-e6af0c4a992b"). InnerVolumeSpecName "kube-api-access-4nk9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:39.123962 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.123924    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:16:39.123962 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.123954    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:16:39.123962 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.123965    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nk9n\" (UniqueName: \"kubernetes.io/projected/34645dd9-5685-4259-a728-e6af0c4a992b-kube-api-access-4nk9n\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:16:39.124225 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.123975    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34645dd9-5685-4259-a728-e6af0c4a992b-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:16:39.124225 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.123983    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34645dd9-5685-4259-a728-e6af0c4a992b-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:16:39.329898 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.329861    2561 generic.go:356] "Generic (PLEG): container finished" podID="34645dd9-5685-4259-a728-e6af0c4a992b" containerID="27492e7dea75a6a43b85c261f14ca5fd75f61895bc19ec55194f8971082e351b" exitCode=0
Sep 16 19:16:39.330119 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.329913    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"
Sep 16 19:16:39.330119 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.329937    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps" event={"ID":"34645dd9-5685-4259-a728-e6af0c4a992b","Type":"ContainerDied","Data":"27492e7dea75a6a43b85c261f14ca5fd75f61895bc19ec55194f8971082e351b"}
Sep 16 19:16:39.330119 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.329983    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps" event={"ID":"34645dd9-5685-4259-a728-e6af0c4a992b","Type":"ContainerDied","Data":"55688ca12a52910fc3ab3de28ba53cd4fd2efaaff7619e17f8918269a9eae89f"}
Sep 16 19:16:39.330119 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.330005    2561 scope.go:117] "RemoveContainer" containerID="27492e7dea75a6a43b85c261f14ca5fd75f61895bc19ec55194f8971082e351b"
Sep 16 19:16:39.340274 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.340249    2561 scope.go:117] "RemoveContainer" containerID="27492e7dea75a6a43b85c261f14ca5fd75f61895bc19ec55194f8971082e351b"
Sep 16 19:16:39.340611 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:16:39.340590    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27492e7dea75a6a43b85c261f14ca5fd75f61895bc19ec55194f8971082e351b\": container with ID starting with 27492e7dea75a6a43b85c261f14ca5fd75f61895bc19ec55194f8971082e351b not found: ID does not exist" containerID="27492e7dea75a6a43b85c261f14ca5fd75f61895bc19ec55194f8971082e351b"
Sep 16 19:16:39.340651 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.340622    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27492e7dea75a6a43b85c261f14ca5fd75f61895bc19ec55194f8971082e351b"} err="failed to get container status \"27492e7dea75a6a43b85c261f14ca5fd75f61895bc19ec55194f8971082e351b\": rpc error: code = NotFound desc = could not find container \"27492e7dea75a6a43b85c261f14ca5fd75f61895bc19ec55194f8971082e351b\": container with ID starting with 27492e7dea75a6a43b85c261f14ca5fd75f61895bc19ec55194f8971082e351b not found: ID does not exist"
Sep 16 19:16:39.357607 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.357564    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"]
Sep 16 19:16:39.367986 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:39.367953    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-mnlps"]
Sep 16 19:16:40.736422 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:40.736390    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34645dd9-5685-4259-a728-e6af0c4a992b" path="/var/lib/kubelet/pods/34645dd9-5685-4259-a728-e6af0c4a992b/volumes"
Sep 16 19:16:41.470523 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:41.470484    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:41.483325 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:41.483297    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:42.359935 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:42.359905    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:16:48.156964 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.156924    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"]
Sep 16 19:16:48.157552 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.157534    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34645dd9-5685-4259-a728-e6af0c4a992b" containerName="main"
Sep 16 19:16:48.157603 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.157555    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="34645dd9-5685-4259-a728-e6af0c4a992b" containerName="main"
Sep 16 19:16:48.157648 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.157618    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="34645dd9-5685-4259-a728-e6af0c4a992b" containerName="main"
Sep 16 19:16:48.162147 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.162125    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.165365 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.165345    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 16 19:16:48.174410 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.174387    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"]
Sep 16 19:16:48.196275 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.196245    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:48.208359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.208331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.208533 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.208365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc98af11-da41-435f-aa45-0c01e26e26bd-tls-certs\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.208533 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.208384    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v76v7\" (UniqueName: \"kubernetes.io/projected/bc98af11-da41-435f-aa45-0c01e26e26bd-kube-api-access-v76v7\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.208663 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.208522    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-home\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.208663 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.208592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-dshm\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.208663 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.208640    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-model-cache\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.309591 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.309555    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-model-cache\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.309806 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.309610    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.309806 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.309639    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc98af11-da41-435f-aa45-0c01e26e26bd-tls-certs\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.309806 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.309671    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v76v7\" (UniqueName: \"kubernetes.io/projected/bc98af11-da41-435f-aa45-0c01e26e26bd-kube-api-access-v76v7\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.309806 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.309751    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-home\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.309806 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.309796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-dshm\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.310124 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.310048    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.310195 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.310121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-model-cache\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.310242 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.310197    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-home\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.312367 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.312331    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-dshm\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.312600 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.312583    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc98af11-da41-435f-aa45-0c01e26e26bd-tls-certs\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.321476 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.321453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v76v7\" (UniqueName: \"kubernetes.io/projected/bc98af11-da41-435f-aa45-0c01e26e26bd-kube-api-access-v76v7\") pod \"stop-v2-kserve-7f68747c45-9x22c\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.474568 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.474529    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:48.756084 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:48.756016    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"]
Sep 16 19:16:48.758366 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:16:48.758335    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc98af11_da41_435f_aa45_0c01e26e26bd.slice/crio-8f1b4f5d621109990f67ce01281e0a39b29bed7de42651dc74b265c90c03af22 WatchSource:0}: Error finding container 8f1b4f5d621109990f67ce01281e0a39b29bed7de42651dc74b265c90c03af22: Status 404 returned error can't find the container with id 8f1b4f5d621109990f67ce01281e0a39b29bed7de42651dc74b265c90c03af22
Sep 16 19:16:49.394262 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:49.394210    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c" event={"ID":"bc98af11-da41-435f-aa45-0c01e26e26bd","Type":"ContainerStarted","Data":"8149c07d475e3f746e9b29faea8242cca8cfa04db899414165560455a8b358e2"}
Sep 16 19:16:49.394778 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:49.394278    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c" event={"ID":"bc98af11-da41-435f-aa45-0c01e26e26bd","Type":"ContainerStarted","Data":"8f1b4f5d621109990f67ce01281e0a39b29bed7de42651dc74b265c90c03af22"}
Sep 16 19:16:49.418453 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:49.418400    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c" podStartSLOduration=1.4183837129999999 podStartE2EDuration="1.418383713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:49.415285208 +0000 UTC m=+1909.218003415" watchObservedRunningTime="2026-09-16 19:16:49.418383713 +0000 UTC m=+1909.221101928"
Sep 16 19:16:49.474664 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:49.474633    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:58.474782 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:58.474725    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:58.487716 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:58.487688    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:16:59.463989 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:16:59.463959    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:17:12.319603 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:12.319564    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:18.418277 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:18.418241    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:18.497922 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:18.497884    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:31.379530 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.379494    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:31.397559 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.397527    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"]
Sep 16 19:17:31.407358 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:17:31.407319    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:17:31.407358 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:31.407358 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:31.407358 ip-10-0-129-125 kubenswrapper[2561]:  > containerID="3cdf7980c3cd058e503bf8d78933af78cd8278008d6c1326cc5bc60ec5db7d0a" cmd=["/bin/sleep","15"]
Sep 16 19:17:31.407626 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.407370    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm" message="" err=<
Sep 16 19:17:31.407626 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:31.407626 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:31.407626 ip-10-0-129-125 kubenswrapper[2561]:  >
Sep 16 19:17:31.407626 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:17:31.407386    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:17:31.407626 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:31.407626 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:31.407626 ip-10-0-129-125 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm" podUID="72095731-e56a-418b-8ed4-1270704bbc6b" containerName="main" containerID="cri-o://3cdf7980c3cd058e503bf8d78933af78cd8278008d6c1326cc5bc60ec5db7d0a"
Sep 16 19:17:31.407626 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.407426    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm" podUID="72095731-e56a-418b-8ed4-1270704bbc6b" containerName="main" containerID="cri-o://3cdf7980c3cd058e503bf8d78933af78cd8278008d6c1326cc5bc60ec5db7d0a" gracePeriod=60
Sep 16 19:17:31.480360 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.480311    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:31.640914 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.640816    2561 generic.go:356] "Generic (PLEG): container finished" podID="72095731-e56a-418b-8ed4-1270704bbc6b" containerID="3cdf7980c3cd058e503bf8d78933af78cd8278008d6c1326cc5bc60ec5db7d0a" exitCode=0
Sep 16 19:17:31.640914 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.640880    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm" event={"ID":"72095731-e56a-418b-8ed4-1270704bbc6b","Type":"ContainerDied","Data":"3cdf7980c3cd058e503bf8d78933af78cd8278008d6c1326cc5bc60ec5db7d0a"}
Sep 16 19:17:31.761075 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.761040    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:17:31.831027 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.830995    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72095731-e56a-418b-8ed4-1270704bbc6b-cabundle-cert\") pod \"72095731-e56a-418b-8ed4-1270704bbc6b\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") "
Sep 16 19:17:31.831237 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831083    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-dshm\") pod \"72095731-e56a-418b-8ed4-1270704bbc6b\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") "
Sep 16 19:17:31.831237 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831119    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-kserve-provision-location\") pod \"72095731-e56a-418b-8ed4-1270704bbc6b\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") "
Sep 16 19:17:31.831237 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831151    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-model-cache\") pod \"72095731-e56a-418b-8ed4-1270704bbc6b\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") "
Sep 16 19:17:31.831237 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831178    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-tmp-dir\") pod \"72095731-e56a-418b-8ed4-1270704bbc6b\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") "
Sep 16 19:17:31.831844 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831225    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr9gg\" (UniqueName: \"kubernetes.io/projected/72095731-e56a-418b-8ed4-1270704bbc6b-kube-api-access-kr9gg\") pod \"72095731-e56a-418b-8ed4-1270704bbc6b\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") "
Sep 16 19:17:31.831844 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831393    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-home\") pod \"72095731-e56a-418b-8ed4-1270704bbc6b\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") "
Sep 16 19:17:31.831844 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831433    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "72095731-e56a-418b-8ed4-1270704bbc6b" (UID: "72095731-e56a-418b-8ed4-1270704bbc6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:31.831844 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831468    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72095731-e56a-418b-8ed4-1270704bbc6b-tls-certs\") pod \"72095731-e56a-418b-8ed4-1270704bbc6b\" (UID: \"72095731-e56a-418b-8ed4-1270704bbc6b\") "
Sep 16 19:17:31.831844 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831523    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72095731-e56a-418b-8ed4-1270704bbc6b" (UID: "72095731-e56a-418b-8ed4-1270704bbc6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:31.831844 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831550    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72095731-e56a-418b-8ed4-1270704bbc6b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "72095731-e56a-418b-8ed4-1270704bbc6b" (UID: "72095731-e56a-418b-8ed4-1270704bbc6b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:31.831844 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831678    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-home" (OuterVolumeSpecName: "home") pod "72095731-e56a-418b-8ed4-1270704bbc6b" (UID: "72095731-e56a-418b-8ed4-1270704bbc6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:31.832314 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831892    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72095731-e56a-418b-8ed4-1270704bbc6b-cabundle-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.832314 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831916    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.832314 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831929    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.832314 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.831940    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.833774 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.833743    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72095731-e56a-418b-8ed4-1270704bbc6b-kube-api-access-kr9gg" (OuterVolumeSpecName: "kube-api-access-kr9gg") pod "72095731-e56a-418b-8ed4-1270704bbc6b" (UID: "72095731-e56a-418b-8ed4-1270704bbc6b"). InnerVolumeSpecName "kube-api-access-kr9gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:31.833896 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.833878    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72095731-e56a-418b-8ed4-1270704bbc6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72095731-e56a-418b-8ed4-1270704bbc6b" (UID: "72095731-e56a-418b-8ed4-1270704bbc6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:31.833946 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.833896    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-dshm" (OuterVolumeSpecName: "dshm") pod "72095731-e56a-418b-8ed4-1270704bbc6b" (UID: "72095731-e56a-418b-8ed4-1270704bbc6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:31.852663 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.852633    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "72095731-e56a-418b-8ed4-1270704bbc6b" (UID: "72095731-e56a-418b-8ed4-1270704bbc6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:31.932738 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.932641    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.932738 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.932681    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72095731-e56a-418b-8ed4-1270704bbc6b-kserve-provision-location\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.932738 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.932695    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr9gg\" (UniqueName: \"kubernetes.io/projected/72095731-e56a-418b-8ed4-1270704bbc6b-kube-api-access-kr9gg\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.932738 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:31.932708    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72095731-e56a-418b-8ed4-1270704bbc6b-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:17:32.649217 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:32.649177    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm" event={"ID":"72095731-e56a-418b-8ed4-1270704bbc6b","Type":"ContainerDied","Data":"daf1795702e73c08eba3068fd5a50e1101e38a24280240b8b8029636ce64516b"}
Sep 16 19:17:32.649217 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:32.649206    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"
Sep 16 19:17:32.649695 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:32.649271    2561 scope.go:117] "RemoveContainer" containerID="3cdf7980c3cd058e503bf8d78933af78cd8278008d6c1326cc5bc60ec5db7d0a"
Sep 16 19:17:32.660327 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:32.660307    2561 scope.go:117] "RemoveContainer" containerID="cd680bfa9ce263fb19232acd0c69eeaea2e9e1527d35e3e0dcb21c886597b1a1"
Sep 16 19:17:32.673340 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:32.673314    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"]
Sep 16 19:17:32.679370 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:32.679340    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-7bwlm"]
Sep 16 19:17:32.736098 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:32.736045    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72095731-e56a-418b-8ed4-1270704bbc6b" path="/var/lib/kubelet/pods/72095731-e56a-418b-8ed4-1270704bbc6b/volumes"
Sep 16 19:17:50.759433 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:50.759397    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"]
Sep 16 19:17:50.795902 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:17:50.795859    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:05.781735 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:05.781692    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c" podUID="bc98af11-da41-435f-aa45-0c01e26e26bd" containerName="main" containerID="cri-o://8149c07d475e3f746e9b29faea8242cca8cfa04db899414165560455a8b358e2" gracePeriod=45
Sep 16 19:18:06.053595 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.053561    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:18:06.060216 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.060192    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v76v7\" (UniqueName: \"kubernetes.io/projected/bc98af11-da41-435f-aa45-0c01e26e26bd-kube-api-access-v76v7\") pod \"bc98af11-da41-435f-aa45-0c01e26e26bd\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") "
Sep 16 19:18:06.060334 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.060258    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc98af11-da41-435f-aa45-0c01e26e26bd-tls-certs\") pod \"bc98af11-da41-435f-aa45-0c01e26e26bd\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") "
Sep 16 19:18:06.060382 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.060356    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-dshm\") pod \"bc98af11-da41-435f-aa45-0c01e26e26bd\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") "
Sep 16 19:18:06.060430 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.060405    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-model-cache\") pod \"bc98af11-da41-435f-aa45-0c01e26e26bd\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") "
Sep 16 19:18:06.060486 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.060447    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-home\") pod \"bc98af11-da41-435f-aa45-0c01e26e26bd\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") "
Sep 16 19:18:06.060486 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.060482    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-tmp-dir\") pod \"bc98af11-da41-435f-aa45-0c01e26e26bd\" (UID: \"bc98af11-da41-435f-aa45-0c01e26e26bd\") "
Sep 16 19:18:06.060727 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.060700    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc98af11-da41-435f-aa45-0c01e26e26bd" (UID: "bc98af11-da41-435f-aa45-0c01e26e26bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:06.060839 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.060806    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-home" (OuterVolumeSpecName: "home") pod "bc98af11-da41-435f-aa45-0c01e26e26bd" (UID: "bc98af11-da41-435f-aa45-0c01e26e26bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:06.060839 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.060816    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc98af11-da41-435f-aa45-0c01e26e26bd" (UID: "bc98af11-da41-435f-aa45-0c01e26e26bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:06.060953 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.060883    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:18:06.062719 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.062694    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-dshm" (OuterVolumeSpecName: "dshm") pod "bc98af11-da41-435f-aa45-0c01e26e26bd" (UID: "bc98af11-da41-435f-aa45-0c01e26e26bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:06.062824 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.062756    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc98af11-da41-435f-aa45-0c01e26e26bd-kube-api-access-v76v7" (OuterVolumeSpecName: "kube-api-access-v76v7") pod "bc98af11-da41-435f-aa45-0c01e26e26bd" (UID: "bc98af11-da41-435f-aa45-0c01e26e26bd"). InnerVolumeSpecName "kube-api-access-v76v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:06.062824 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.062756    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc98af11-da41-435f-aa45-0c01e26e26bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc98af11-da41-435f-aa45-0c01e26e26bd" (UID: "bc98af11-da41-435f-aa45-0c01e26e26bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:06.161389 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.161352    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:18:06.161389 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.161381    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:18:06.161389 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.161394    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v76v7\" (UniqueName: \"kubernetes.io/projected/bc98af11-da41-435f-aa45-0c01e26e26bd-kube-api-access-v76v7\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:18:06.161644 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.161404    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc98af11-da41-435f-aa45-0c01e26e26bd-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:18:06.161644 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.161412    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc98af11-da41-435f-aa45-0c01e26e26bd-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:18:06.841408 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.841360    2561 generic.go:356] "Generic (PLEG): container finished" podID="bc98af11-da41-435f-aa45-0c01e26e26bd" containerID="8149c07d475e3f746e9b29faea8242cca8cfa04db899414165560455a8b358e2" exitCode=0
Sep 16 19:18:06.841908 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.841419    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"
Sep 16 19:18:06.841908 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.841438    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c" event={"ID":"bc98af11-da41-435f-aa45-0c01e26e26bd","Type":"ContainerDied","Data":"8149c07d475e3f746e9b29faea8242cca8cfa04db899414165560455a8b358e2"}
Sep 16 19:18:06.841908 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.841487    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c" event={"ID":"bc98af11-da41-435f-aa45-0c01e26e26bd","Type":"ContainerDied","Data":"8f1b4f5d621109990f67ce01281e0a39b29bed7de42651dc74b265c90c03af22"}
Sep 16 19:18:06.841908 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.841510    2561 scope.go:117] "RemoveContainer" containerID="8149c07d475e3f746e9b29faea8242cca8cfa04db899414165560455a8b358e2"
Sep 16 19:18:06.850930 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.850908    2561 scope.go:117] "RemoveContainer" containerID="8149c07d475e3f746e9b29faea8242cca8cfa04db899414165560455a8b358e2"
Sep 16 19:18:06.851215 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:18:06.851195    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8149c07d475e3f746e9b29faea8242cca8cfa04db899414165560455a8b358e2\": container with ID starting with 8149c07d475e3f746e9b29faea8242cca8cfa04db899414165560455a8b358e2 not found: ID does not exist" containerID="8149c07d475e3f746e9b29faea8242cca8cfa04db899414165560455a8b358e2"
Sep 16 19:18:06.851294 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.851223    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8149c07d475e3f746e9b29faea8242cca8cfa04db899414165560455a8b358e2"} err="failed to get container status \"8149c07d475e3f746e9b29faea8242cca8cfa04db899414165560455a8b358e2\": rpc error: code = NotFound desc = could not find container \"8149c07d475e3f746e9b29faea8242cca8cfa04db899414165560455a8b358e2\": container with ID starting with 8149c07d475e3f746e9b29faea8242cca8cfa04db899414165560455a8b358e2 not found: ID does not exist"
Sep 16 19:18:06.861836 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.861812    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"]
Sep 16 19:18:06.868880 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:06.868853    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x22c"]
Sep 16 19:18:08.734852 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:08.734813    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc98af11-da41-435f-aa45-0c01e26e26bd" path="/var/lib/kubelet/pods/bc98af11-da41-435f-aa45-0c01e26e26bd/volumes"
Sep 16 19:18:12.553473 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:12.553439    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:12.653374 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:12.653341    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:17.507369 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:17.507334    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"]
Sep 16 19:18:17.568581 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:17.568522    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:20.327925 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:20.327889    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:20.439929 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:20.439842    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:23.856363 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:23.856316    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:27.213300 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:27.213264    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:27.412428 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:27.412391    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:32.527722 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.527643    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh" podUID="4d2527d2-9191-4b1c-8ac3-49908ee1889c" containerName="main" containerID="cri-o://2e27c7074297bb6bdd777dc083f62eb718e5a6669c8f27980462cd9ef4f90d1e" gracePeriod=45
Sep 16 19:18:32.784887 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.784820    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:18:32.887126 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.887091    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2527d2-9191-4b1c-8ac3-49908ee1889c-tls-certs\") pod \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") "
Sep 16 19:18:32.887304 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.887149    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kqrw\" (UniqueName: \"kubernetes.io/projected/4d2527d2-9191-4b1c-8ac3-49908ee1889c-kube-api-access-6kqrw\") pod \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") "
Sep 16 19:18:32.887304 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.887185    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-tmp-dir\") pod \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") "
Sep 16 19:18:32.887304 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.887255    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-home\") pod \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") "
Sep 16 19:18:32.887304 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.887301    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-dshm\") pod \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") "
Sep 16 19:18:32.887482 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.887369    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-model-cache\") pod \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\" (UID: \"4d2527d2-9191-4b1c-8ac3-49908ee1889c\") "
Sep 16 19:18:32.887541 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.887501    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-home" (OuterVolumeSpecName: "home") pod "4d2527d2-9191-4b1c-8ac3-49908ee1889c" (UID: "4d2527d2-9191-4b1c-8ac3-49908ee1889c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:32.887605 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.887548    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d2527d2-9191-4b1c-8ac3-49908ee1889c" (UID: "4d2527d2-9191-4b1c-8ac3-49908ee1889c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:32.887758 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.887732    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d2527d2-9191-4b1c-8ac3-49908ee1889c" (UID: "4d2527d2-9191-4b1c-8ac3-49908ee1889c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:32.887758 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.887752    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:18:32.887867 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.887772    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:18:32.889441 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.889419    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-dshm" (OuterVolumeSpecName: "dshm") pod "4d2527d2-9191-4b1c-8ac3-49908ee1889c" (UID: "4d2527d2-9191-4b1c-8ac3-49908ee1889c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:32.889517 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.889501    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d2527d2-9191-4b1c-8ac3-49908ee1889c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d2527d2-9191-4b1c-8ac3-49908ee1889c" (UID: "4d2527d2-9191-4b1c-8ac3-49908ee1889c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:32.889581 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.889561    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2527d2-9191-4b1c-8ac3-49908ee1889c-kube-api-access-6kqrw" (OuterVolumeSpecName: "kube-api-access-6kqrw") pod "4d2527d2-9191-4b1c-8ac3-49908ee1889c" (UID: "4d2527d2-9191-4b1c-8ac3-49908ee1889c"). InnerVolumeSpecName "kube-api-access-6kqrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:32.988295 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.988259    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:18:32.988295 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.988289    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2527d2-9191-4b1c-8ac3-49908ee1889c-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:18:32.988295 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.988303    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kqrw\" (UniqueName: \"kubernetes.io/projected/4d2527d2-9191-4b1c-8ac3-49908ee1889c-kube-api-access-6kqrw\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:18:32.988562 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.988313    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2527d2-9191-4b1c-8ac3-49908ee1889c-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:18:32.988914 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.988888    2561 generic.go:356] "Generic (PLEG): container finished" podID="4d2527d2-9191-4b1c-8ac3-49908ee1889c" containerID="2e27c7074297bb6bdd777dc083f62eb718e5a6669c8f27980462cd9ef4f90d1e" exitCode=0
Sep 16 19:18:32.989007 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.988933    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"
Sep 16 19:18:32.989007 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.988977    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh" event={"ID":"4d2527d2-9191-4b1c-8ac3-49908ee1889c","Type":"ContainerDied","Data":"2e27c7074297bb6bdd777dc083f62eb718e5a6669c8f27980462cd9ef4f90d1e"}
Sep 16 19:18:32.989112 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.989032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh" event={"ID":"4d2527d2-9191-4b1c-8ac3-49908ee1889c","Type":"ContainerDied","Data":"ef69f7ddac57906c5ee70507fd7a4a2c62ee0833e3df9222315b0852d5fb6430"}
Sep 16 19:18:32.989112 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.989074    2561 scope.go:117] "RemoveContainer" containerID="2e27c7074297bb6bdd777dc083f62eb718e5a6669c8f27980462cd9ef4f90d1e"
Sep 16 19:18:32.999111 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.999092    2561 scope.go:117] "RemoveContainer" containerID="2e27c7074297bb6bdd777dc083f62eb718e5a6669c8f27980462cd9ef4f90d1e"
Sep 16 19:18:32.999377 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:18:32.999357    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e27c7074297bb6bdd777dc083f62eb718e5a6669c8f27980462cd9ef4f90d1e\": container with ID starting with 2e27c7074297bb6bdd777dc083f62eb718e5a6669c8f27980462cd9ef4f90d1e not found: ID does not exist" containerID="2e27c7074297bb6bdd777dc083f62eb718e5a6669c8f27980462cd9ef4f90d1e"
Sep 16 19:18:32.999443 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:32.999386    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e27c7074297bb6bdd777dc083f62eb718e5a6669c8f27980462cd9ef4f90d1e"} err="failed to get container status \"2e27c7074297bb6bdd777dc083f62eb718e5a6669c8f27980462cd9ef4f90d1e\": rpc error: code = NotFound desc = could not find container \"2e27c7074297bb6bdd777dc083f62eb718e5a6669c8f27980462cd9ef4f90d1e\": container with ID starting with 2e27c7074297bb6bdd777dc083f62eb718e5a6669c8f27980462cd9ef4f90d1e not found: ID does not exist"
Sep 16 19:18:33.044425 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:33.044342    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"]
Sep 16 19:18:33.051036 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:33.051010    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-dw5gh"]
Sep 16 19:18:33.736388 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:33.736352    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:33.837654 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:33.837618    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:34.039745 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:34.039655    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:34.140225 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:34.140183    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:34.734456 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:34.734421    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d2527d2-9191-4b1c-8ac3-49908ee1889c" path="/var/lib/kubelet/pods/4d2527d2-9191-4b1c-8ac3-49908ee1889c/volumes"
Sep 16 19:18:38.058231 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:38.058194    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:49.373981 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.373949    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"]
Sep 16 19:18:49.374453 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.374438    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc98af11-da41-435f-aa45-0c01e26e26bd" containerName="main"
Sep 16 19:18:49.374502 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.374458    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc98af11-da41-435f-aa45-0c01e26e26bd" containerName="main"
Sep 16 19:18:49.374502 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.374487    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72095731-e56a-418b-8ed4-1270704bbc6b" containerName="storage-initializer"
Sep 16 19:18:49.374502 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.374496    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="72095731-e56a-418b-8ed4-1270704bbc6b" containerName="storage-initializer"
Sep 16 19:18:49.374596 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.374502    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72095731-e56a-418b-8ed4-1270704bbc6b" containerName="main"
Sep 16 19:18:49.374596 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.374508    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="72095731-e56a-418b-8ed4-1270704bbc6b" containerName="main"
Sep 16 19:18:49.374596 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.374516    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d2527d2-9191-4b1c-8ac3-49908ee1889c" containerName="main"
Sep 16 19:18:49.374596 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.374520    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d2527d2-9191-4b1c-8ac3-49908ee1889c" containerName="main"
Sep 16 19:18:49.374596 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.374582    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d2527d2-9191-4b1c-8ac3-49908ee1889c" containerName="main"
Sep 16 19:18:49.374596 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.374592    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc98af11-da41-435f-aa45-0c01e26e26bd" containerName="main"
Sep 16 19:18:49.374596 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.374597    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="72095731-e56a-418b-8ed4-1270704bbc6b" containerName="main"
Sep 16 19:18:49.379245 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.379220    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.382305 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.382277    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 19:18:49.383816 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.383582    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-qkkc7\""
Sep 16 19:18:49.383816 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.383614    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 19:18:49.383816 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.383655    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:18:49.383816 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.383588    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 19:18:49.384081 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.383979    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 19:18:49.391915 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.391895    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"]
Sep 16 19:18:49.415151 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.415122    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:49.439318 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.439283    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-tls-certs\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.439318 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.439321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-home\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.439553 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.439347    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.439553 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.439444    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-dshm\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.439553 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.439479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-model-cache\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.439553 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.439506    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.439553 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.439548    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxrc9\" (UniqueName: \"kubernetes.io/projected/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-kube-api-access-cxrc9\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.439726 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.439576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-tmp-dir\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.540794 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.540749    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.540981 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.540806    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrc9\" (UniqueName: \"kubernetes.io/projected/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-kube-api-access-cxrc9\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.540981 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.540838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-tmp-dir\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.540981 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.540862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-tls-certs\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.540981 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.540879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-home\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.540981 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.540899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.540981 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.540942    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-dshm\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.540981 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.540970    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-model-cache\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.541423 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.541254    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.541423 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.541292    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-tmp-dir\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.541529 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.541453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-home\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.541595 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.541572    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-model-cache\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.541664 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.541643    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.543452 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.543433    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-dshm\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.543614 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.543599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-tls-certs\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.550624 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.550600    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrc9\" (UniqueName: \"kubernetes.io/projected/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-kube-api-access-cxrc9\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.650816 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.650696    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"]
Sep 16 19:18:49.655812 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.655786    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"
Sep 16 19:18:49.659204 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.659177    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-tv2wh\""
Sep 16 19:18:49.665271 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.665090    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"]
Sep 16 19:18:49.678257 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.678221    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:49.691311 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.691276    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:49.744238 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.743977    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8r5k\" (UniqueName: \"kubernetes.io/projected/87ac50e7-fefa-4592-b3d9-e9ef4d388924-kube-api-access-p8r5k\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s\" (UID: \"87ac50e7-fefa-4592-b3d9-e9ef4d388924\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"
Sep 16 19:18:49.744238 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.744104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87ac50e7-fefa-4592-b3d9-e9ef4d388924-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s\" (UID: \"87ac50e7-fefa-4592-b3d9-e9ef4d388924\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"
Sep 16 19:18:49.775226 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.775192    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:49.845075 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.845019    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8r5k\" (UniqueName: \"kubernetes.io/projected/87ac50e7-fefa-4592-b3d9-e9ef4d388924-kube-api-access-p8r5k\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s\" (UID: \"87ac50e7-fefa-4592-b3d9-e9ef4d388924\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"
Sep 16 19:18:49.845286 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.845122    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87ac50e7-fefa-4592-b3d9-e9ef4d388924-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s\" (UID: \"87ac50e7-fefa-4592-b3d9-e9ef4d388924\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"
Sep 16 19:18:49.847843 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.847814    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87ac50e7-fefa-4592-b3d9-e9ef4d388924-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s\" (UID: \"87ac50e7-fefa-4592-b3d9-e9ef4d388924\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"
Sep 16 19:18:49.855219 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.855190    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8r5k\" (UniqueName: \"kubernetes.io/projected/87ac50e7-fefa-4592-b3d9-e9ef4d388924-kube-api-access-p8r5k\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s\" (UID: \"87ac50e7-fefa-4592-b3d9-e9ef4d388924\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"
Sep 16 19:18:49.968835 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.968787    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"
Sep 16 19:18:49.988163 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:49.988135    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"]
Sep 16 19:18:49.989714 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:18:49.989679    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbbe55b_64aa_4b22_8e6d_6be860b17d58.slice/crio-08194f06ec389bf7452c9c1ae6ef5d574862e699e11d4a3581817169bfbe3fc6 WatchSource:0}: Error finding container 08194f06ec389bf7452c9c1ae6ef5d574862e699e11d4a3581817169bfbe3fc6: Status 404 returned error can't find the container with id 08194f06ec389bf7452c9c1ae6ef5d574862e699e11d4a3581817169bfbe3fc6
Sep 16 19:18:50.093601 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:50.093558    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb" event={"ID":"cdbbe55b-64aa-4b22-8e6d-6be860b17d58","Type":"ContainerStarted","Data":"5a7ee2cde89abe26d029419195e5c0397bd79ed2861f65fdecc69d3950098e8f"}
Sep 16 19:18:50.093782 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:50.093616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb" event={"ID":"cdbbe55b-64aa-4b22-8e6d-6be860b17d58","Type":"ContainerStarted","Data":"08194f06ec389bf7452c9c1ae6ef5d574862e699e11d4a3581817169bfbe3fc6"}
Sep 16 19:18:50.258168 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:50.258134    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"]
Sep 16 19:18:50.259343 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:18:50.259310    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ac50e7_fefa_4592_b3d9_e9ef4d388924.slice/crio-c1f0cccf5b562c4525bb9336cdd4e49c5f05e1f8d894cce4d65e3d14b2e9e16b WatchSource:0}: Error finding container c1f0cccf5b562c4525bb9336cdd4e49c5f05e1f8d894cce4d65e3d14b2e9e16b: Status 404 returned error can't find the container with id c1f0cccf5b562c4525bb9336cdd4e49c5f05e1f8d894cce4d65e3d14b2e9e16b
Sep 16 19:18:51.104896 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:51.104848    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s" event={"ID":"87ac50e7-fefa-4592-b3d9-e9ef4d388924","Type":"ContainerStarted","Data":"7e05d8e0fc132331f79b64e308f56196357381d89574120de06ce9e604d47dfb"}
Sep 16 19:18:51.105413 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:51.104914    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s" event={"ID":"87ac50e7-fefa-4592-b3d9-e9ef4d388924","Type":"ContainerStarted","Data":"c1f0cccf5b562c4525bb9336cdd4e49c5f05e1f8d894cce4d65e3d14b2e9e16b"}
Sep 16 19:18:51.105413 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:51.105067    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"
Sep 16 19:18:51.138561 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:51.138472    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s" podStartSLOduration=2.1384526680000002 podStartE2EDuration="2.138452668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:51.137591276 +0000 UTC m=+2030.940309490" watchObservedRunningTime="2026-09-16 19:18:51.138452668 +0000 UTC m=+2030.941170895"
Sep 16 19:18:53.119814 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:53.119776    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdbbe55b-64aa-4b22-8e6d-6be860b17d58" containerID="5a7ee2cde89abe26d029419195e5c0397bd79ed2861f65fdecc69d3950098e8f" exitCode=0
Sep 16 19:18:53.120362 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:53.119853    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb" event={"ID":"cdbbe55b-64aa-4b22-8e6d-6be860b17d58","Type":"ContainerDied","Data":"5a7ee2cde89abe26d029419195e5c0397bd79ed2861f65fdecc69d3950098e8f"}
Sep 16 19:18:54.127069 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:54.127032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb" event={"ID":"cdbbe55b-64aa-4b22-8e6d-6be860b17d58","Type":"ContainerStarted","Data":"9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7"}
Sep 16 19:18:54.153471 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:54.153413    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb" podStartSLOduration=2.631376915 podStartE2EDuration="5.153395748s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.522018833s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:54.15130403 +0000 UTC m=+2033.954022247" watchObservedRunningTime="2026-09-16 19:18:54.153395748 +0000 UTC m=+2033.956113964"
Sep 16 19:18:54.668226 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:54.668191    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:54.691911 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:54.691877    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:59.692355 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:59.692314    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:18:59.705300 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:18:59.705268    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:19:00.173947 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:00.173915    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:19:00.931453 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:00.931412    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:01.032277 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:01.032242    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:16.722388 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:16.722341    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:22.114805 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:22.114716    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"
Sep 16 19:19:28.748846 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:28.748810    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:28.789303 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:28.789130    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"]
Sep 16 19:19:28.799228 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:28.799199    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"]
Sep 16 19:19:28.810935 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:19:28.810888    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:19:28.810935 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:19:28.810935 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:19:28.810935 ip-10-0-129-125 kubenswrapper[2561]:  > containerID="9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7" cmd=["/bin/sleep","15"]
Sep 16 19:19:28.811303 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:28.810957    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb" message="" err=<
Sep 16 19:19:28.811303 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:19:28.811303 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:19:28.811303 ip-10-0-129-125 kubenswrapper[2561]:  >
Sep 16 19:19:28.811303 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:19:28.810973    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:19:28.811303 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:19:28.811303 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:19:28.811303 ip-10-0-129-125 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb" podUID="cdbbe55b-64aa-4b22-8e6d-6be860b17d58" containerName="main" containerID="cri-o://9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7"
Sep 16 19:19:28.811303 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:28.811032    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb" podUID="cdbbe55b-64aa-4b22-8e6d-6be860b17d58" containerName="main" containerID="cri-o://9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7" gracePeriod=60
Sep 16 19:19:28.847186 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:28.847156    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:29.162433 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.162406    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:19:29.187021 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.186986    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-kserve-provision-location\") pod \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") "
Sep 16 19:19:29.187240 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.187037    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-home\") pod \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") "
Sep 16 19:19:29.187240 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.187107    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxrc9\" (UniqueName: \"kubernetes.io/projected/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-kube-api-access-cxrc9\") pod \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") "
Sep 16 19:19:29.187240 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.187167    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-dshm\") pod \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") "
Sep 16 19:19:29.187427 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.187268    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-tmp-dir\") pod \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") "
Sep 16 19:19:29.187427 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.187379    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-home" (OuterVolumeSpecName: "home") pod "cdbbe55b-64aa-4b22-8e6d-6be860b17d58" (UID: "cdbbe55b-64aa-4b22-8e6d-6be860b17d58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.187427 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.187398    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-cabundle-cert\") pod \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") "
Sep 16 19:19:29.187580 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.187437    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-tls-certs\") pod \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") "
Sep 16 19:19:29.187580 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.187491    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-model-cache\") pod \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\" (UID: \"cdbbe55b-64aa-4b22-8e6d-6be860b17d58\") "
Sep 16 19:19:29.187704 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.187604    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cdbbe55b-64aa-4b22-8e6d-6be860b17d58" (UID: "cdbbe55b-64aa-4b22-8e6d-6be860b17d58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.187761 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.187724    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cdbbe55b-64aa-4b22-8e6d-6be860b17d58" (UID: "cdbbe55b-64aa-4b22-8e6d-6be860b17d58"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:29.188028 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.187973    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.188028 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.188000    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.188209 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.188040    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-cabundle-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.188209 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.188084    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-model-cache" (OuterVolumeSpecName: "model-cache") pod "cdbbe55b-64aa-4b22-8e6d-6be860b17d58" (UID: "cdbbe55b-64aa-4b22-8e6d-6be860b17d58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.190049 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.190010    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-kube-api-access-cxrc9" (OuterVolumeSpecName: "kube-api-access-cxrc9") pod "cdbbe55b-64aa-4b22-8e6d-6be860b17d58" (UID: "cdbbe55b-64aa-4b22-8e6d-6be860b17d58"). InnerVolumeSpecName "kube-api-access-cxrc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:29.190911 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.190880    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-dshm" (OuterVolumeSpecName: "dshm") pod "cdbbe55b-64aa-4b22-8e6d-6be860b17d58" (UID: "cdbbe55b-64aa-4b22-8e6d-6be860b17d58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.190911 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.190894    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdbbe55b-64aa-4b22-8e6d-6be860b17d58" (UID: "cdbbe55b-64aa-4b22-8e6d-6be860b17d58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:29.213142 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.213104    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cdbbe55b-64aa-4b22-8e6d-6be860b17d58" (UID: "cdbbe55b-64aa-4b22-8e6d-6be860b17d58"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.288625 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.288542    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxrc9\" (UniqueName: \"kubernetes.io/projected/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-kube-api-access-cxrc9\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.288625 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.288572    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.288625 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.288582    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.288625 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.288591    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.288625 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.288599    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdbbe55b-64aa-4b22-8e6d-6be860b17d58-kserve-provision-location\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.330860 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.330821    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdbbe55b-64aa-4b22-8e6d-6be860b17d58" containerID="9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7" exitCode=0
Sep 16 19:19:29.331070 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.330878    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"
Sep 16 19:19:29.331070 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.330902    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb" event={"ID":"cdbbe55b-64aa-4b22-8e6d-6be860b17d58","Type":"ContainerDied","Data":"9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7"}
Sep 16 19:19:29.331070 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.330959    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb" event={"ID":"cdbbe55b-64aa-4b22-8e6d-6be860b17d58","Type":"ContainerDied","Data":"08194f06ec389bf7452c9c1ae6ef5d574862e699e11d4a3581817169bfbe3fc6"}
Sep 16 19:19:29.331070 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.330989    2561 scope.go:117] "RemoveContainer" containerID="9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7"
Sep 16 19:19:29.342270 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.342249    2561 scope.go:117] "RemoveContainer" containerID="5a7ee2cde89abe26d029419195e5c0397bd79ed2861f65fdecc69d3950098e8f"
Sep 16 19:19:29.350809 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.350788    2561 scope.go:117] "RemoveContainer" containerID="9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7"
Sep 16 19:19:29.351091 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:19:29.351070    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7\": container with ID starting with 9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7 not found: ID does not exist" containerID="9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7"
Sep 16 19:19:29.351154 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.351104    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7"} err="failed to get container status \"9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7\": rpc error: code = NotFound desc = could not find container \"9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7\": container with ID starting with 9fd89f721cb1b46f6ca287a3b933d2b6324d0f219d4ffed5277b087f4b0094d7 not found: ID does not exist"
Sep 16 19:19:29.351154 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.351125    2561 scope.go:117] "RemoveContainer" containerID="5a7ee2cde89abe26d029419195e5c0397bd79ed2861f65fdecc69d3950098e8f"
Sep 16 19:19:29.351396 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:19:29.351376    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7ee2cde89abe26d029419195e5c0397bd79ed2861f65fdecc69d3950098e8f\": container with ID starting with 5a7ee2cde89abe26d029419195e5c0397bd79ed2861f65fdecc69d3950098e8f not found: ID does not exist" containerID="5a7ee2cde89abe26d029419195e5c0397bd79ed2861f65fdecc69d3950098e8f"
Sep 16 19:19:29.351458 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.351405    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7ee2cde89abe26d029419195e5c0397bd79ed2861f65fdecc69d3950098e8f"} err="failed to get container status \"5a7ee2cde89abe26d029419195e5c0397bd79ed2861f65fdecc69d3950098e8f\": rpc error: code = NotFound desc = could not find container \"5a7ee2cde89abe26d029419195e5c0397bd79ed2861f65fdecc69d3950098e8f\": container with ID starting with 5a7ee2cde89abe26d029419195e5c0397bd79ed2861f65fdecc69d3950098e8f not found: ID does not exist"
Sep 16 19:19:29.358739 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.358716    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"]
Sep 16 19:19:29.362478 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:29.362454    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-q5bwb"]
Sep 16 19:19:30.735878 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:30.735848    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdbbe55b-64aa-4b22-8e6d-6be860b17d58" path="/var/lib/kubelet/pods/cdbbe55b-64aa-4b22-8e6d-6be860b17d58/volumes"
Sep 16 19:19:32.115098 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:32.115026    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s" podUID="87ac50e7-fefa-4592-b3d9-e9ef4d388924" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:19:39.997414 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:39.997370    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:42.114924 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:42.114877    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s" podUID="87ac50e7-fefa-4592-b3d9-e9ef4d388924" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:19:43.816935 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:43.816894    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s" podUID="87ac50e7-fefa-4592-b3d9-e9ef4d388924" containerName="main" containerID="cri-o://7e05d8e0fc132331f79b64e308f56196357381d89574120de06ce9e604d47dfb" gracePeriod=45
Sep 16 19:19:52.115615 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:52.115569    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s" podUID="87ac50e7-fefa-4592-b3d9-e9ef4d388924" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:19:52.116046 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:52.115725    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"
Sep 16 19:19:56.143670 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:56.143634    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:56.303377 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:56.303345    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:56.425337 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:19:56.425259    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:00.899270 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:00.899243    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:20:00.904183 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:00.904157    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:20:01.175653 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:01.175542    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:02.115225 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:02.115180    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s" podUID="87ac50e7-fefa-4592-b3d9-e9ef4d388924" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:20:02.492091 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:02.492031    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:12.114648 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:12.114609    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s" podUID="87ac50e7-fefa-4592-b3d9-e9ef4d388924" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:20:14.076915 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.076853    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"
Sep 16 19:20:14.100072 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.100032    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87ac50e7-fefa-4592-b3d9-e9ef4d388924-tls-certs\") pod \"87ac50e7-fefa-4592-b3d9-e9ef4d388924\" (UID: \"87ac50e7-fefa-4592-b3d9-e9ef4d388924\") "
Sep 16 19:20:14.100258 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.100085    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8r5k\" (UniqueName: \"kubernetes.io/projected/87ac50e7-fefa-4592-b3d9-e9ef4d388924-kube-api-access-p8r5k\") pod \"87ac50e7-fefa-4592-b3d9-e9ef4d388924\" (UID: \"87ac50e7-fefa-4592-b3d9-e9ef4d388924\") "
Sep 16 19:20:14.102461 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.102436    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ac50e7-fefa-4592-b3d9-e9ef4d388924-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87ac50e7-fefa-4592-b3d9-e9ef4d388924" (UID: "87ac50e7-fefa-4592-b3d9-e9ef4d388924"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:14.102650 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.102633    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ac50e7-fefa-4592-b3d9-e9ef4d388924-kube-api-access-p8r5k" (OuterVolumeSpecName: "kube-api-access-p8r5k") pod "87ac50e7-fefa-4592-b3d9-e9ef4d388924" (UID: "87ac50e7-fefa-4592-b3d9-e9ef4d388924"). InnerVolumeSpecName "kube-api-access-p8r5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:14.201209 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.201151    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87ac50e7-fefa-4592-b3d9-e9ef4d388924-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:20:14.201209 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.201200    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8r5k\" (UniqueName: \"kubernetes.io/projected/87ac50e7-fefa-4592-b3d9-e9ef4d388924-kube-api-access-p8r5k\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:20:14.584771 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.584738    2561 generic.go:356] "Generic (PLEG): container finished" podID="87ac50e7-fefa-4592-b3d9-e9ef4d388924" containerID="7e05d8e0fc132331f79b64e308f56196357381d89574120de06ce9e604d47dfb" exitCode=0
Sep 16 19:20:14.584984 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.584782    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"
Sep 16 19:20:14.584984 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.584787    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s" event={"ID":"87ac50e7-fefa-4592-b3d9-e9ef4d388924","Type":"ContainerDied","Data":"7e05d8e0fc132331f79b64e308f56196357381d89574120de06ce9e604d47dfb"}
Sep 16 19:20:14.584984 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.584915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s" event={"ID":"87ac50e7-fefa-4592-b3d9-e9ef4d388924","Type":"ContainerDied","Data":"c1f0cccf5b562c4525bb9336cdd4e49c5f05e1f8d894cce4d65e3d14b2e9e16b"}
Sep 16 19:20:14.584984 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.584948    2561 scope.go:117] "RemoveContainer" containerID="7e05d8e0fc132331f79b64e308f56196357381d89574120de06ce9e604d47dfb"
Sep 16 19:20:14.595241 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.595219    2561 scope.go:117] "RemoveContainer" containerID="7e05d8e0fc132331f79b64e308f56196357381d89574120de06ce9e604d47dfb"
Sep 16 19:20:14.595549 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:20:14.595525    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e05d8e0fc132331f79b64e308f56196357381d89574120de06ce9e604d47dfb\": container with ID starting with 7e05d8e0fc132331f79b64e308f56196357381d89574120de06ce9e604d47dfb not found: ID does not exist" containerID="7e05d8e0fc132331f79b64e308f56196357381d89574120de06ce9e604d47dfb"
Sep 16 19:20:14.595618 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.595561    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e05d8e0fc132331f79b64e308f56196357381d89574120de06ce9e604d47dfb"} err="failed to get container status \"7e05d8e0fc132331f79b64e308f56196357381d89574120de06ce9e604d47dfb\": rpc error: code = NotFound desc = could not find container \"7e05d8e0fc132331f79b64e308f56196357381d89574120de06ce9e604d47dfb\": container with ID starting with 7e05d8e0fc132331f79b64e308f56196357381d89574120de06ce9e604d47dfb not found: ID does not exist"
Sep 16 19:20:14.610582 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.610535    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"]
Sep 16 19:20:14.613206 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.613180    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9vd5s"]
Sep 16 19:20:14.734043 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:14.734010    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87ac50e7-fefa-4592-b3d9-e9ef4d388924" path="/var/lib/kubelet/pods/87ac50e7-fefa-4592-b3d9-e9ef4d388924/volumes"
Sep 16 19:20:18.045672 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:18.045632    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:18.197869 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:18.197832    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:25.292859 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:25.292791    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:30.026651 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.026611    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"]
Sep 16 19:20:30.027047 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.027026    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ac50e7-fefa-4592-b3d9-e9ef4d388924" containerName="main"
Sep 16 19:20:30.027047 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.027038    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ac50e7-fefa-4592-b3d9-e9ef4d388924" containerName="main"
Sep 16 19:20:30.027152 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.027049    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbbe55b-64aa-4b22-8e6d-6be860b17d58" containerName="storage-initializer"
Sep 16 19:20:30.027152 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.027071    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbbe55b-64aa-4b22-8e6d-6be860b17d58" containerName="storage-initializer"
Sep 16 19:20:30.027152 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.027081    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbbe55b-64aa-4b22-8e6d-6be860b17d58" containerName="main"
Sep 16 19:20:30.027152 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.027087    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbbe55b-64aa-4b22-8e6d-6be860b17d58" containerName="main"
Sep 16 19:20:30.027152 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.027143    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ac50e7-fefa-4592-b3d9-e9ef4d388924" containerName="main"
Sep 16 19:20:30.027152 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.027155    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdbbe55b-64aa-4b22-8e6d-6be860b17d58" containerName="main"
Sep 16 19:20:30.031739 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.031721    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.045697 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:20:30.045659    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-125.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-leave-group-5b69d57a\": no relationship found between node 'ip-10-0-129-125.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 16 19:20:30.045904 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:20:30.045712    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-prkt7\" is forbidden: User \"system:node:ip-10-0-129-125.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-leave-group-5b69d57a\": no relationship found between node 'ip-10-0-129-125.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-prkt7\"" type="*v1.Secret"
Sep 16 19:20:30.045904 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:20:30.045726    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"leave-v2-kserve-556b4dddf9-wrc2z\" is forbidden: User \"system:node:ip-10-0-129-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"e2e-test-leave-group-5b69d57a\": no relationship found between node 'ip-10-0-129-125.ec2.internal' and this object" podUID="dbc82e67-1421-4ff2-bd09-426c985addbd" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.045904 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:20:30.045720    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"leave-v2-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-129-125.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-leave-group-5b69d57a\": no relationship found between node 'ip-10-0-129-125.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 16 19:20:30.045904 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:20:30.045748    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-125.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-leave-group-5b69d57a\": no relationship found between node 'ip-10-0-129-125.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 16 19:20:30.055240 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.055209    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"]
Sep 16 19:20:30.131376 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.131337    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:30.155514 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.155474    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-dshm\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.155514 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.155512    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.155744 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.155577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc82e67-1421-4ff2-bd09-426c985addbd-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.155744 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.155622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-model-cache\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.155744 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.155690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvth7\" (UniqueName: \"kubernetes.io/projected/dbc82e67-1421-4ff2-bd09-426c985addbd-kube-api-access-lvth7\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.155842 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.155762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-home\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.256790 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.256752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-home\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.256956 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.256801    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-dshm\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.256956 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.256818    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.257107 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.256951    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc82e67-1421-4ff2-bd09-426c985addbd-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.257107 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.256997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-model-cache\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.257107 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.257044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvth7\" (UniqueName: \"kubernetes.io/projected/dbc82e67-1421-4ff2-bd09-426c985addbd-kube-api-access-lvth7\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.257276 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.257211    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.257276 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.257264    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-home\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.257388 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.257321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-model-cache\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.259307 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.259270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-dshm\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.938128 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.938085    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 16 19:20:30.950752 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.950715    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc82e67-1421-4ff2-bd09-426c985addbd-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:30.969111 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:30.969073    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 16 19:20:31.275674 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:20:31.275637    2561 projected.go:291] Couldn't get configMap e2e-test-leave-group-5b69d57a/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 16 19:20:31.275674 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:20:31.275679    2561 projected.go:196] Error preparing data for projected volume kube-api-access-lvth7 for pod e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z: failed to sync configmap cache: timed out waiting for the condition
Sep 16 19:20:31.276179 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:20:31.275736    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbc82e67-1421-4ff2-bd09-426c985addbd-kube-api-access-lvth7 podName:dbc82e67-1421-4ff2-bd09-426c985addbd nodeName:}" failed. No retries permitted until 2026-09-16 19:20:31.77571943 +0000 UTC m=+2131.578437624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lvth7" (UniqueName: "kubernetes.io/projected/dbc82e67-1421-4ff2-bd09-426c985addbd-kube-api-access-lvth7") pod "leave-v2-kserve-556b4dddf9-wrc2z" (UID: "dbc82e67-1421-4ff2-bd09-426c985addbd") : failed to sync configmap cache: timed out waiting for the condition
Sep 16 19:20:31.289913 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:31.289883    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-prkt7\""
Sep 16 19:20:31.430361 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:31.430326    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 16 19:20:31.873901 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:31.873857    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvth7\" (UniqueName: \"kubernetes.io/projected/dbc82e67-1421-4ff2-bd09-426c985addbd-kube-api-access-lvth7\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:31.876560 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:31.876534    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvth7\" (UniqueName: \"kubernetes.io/projected/dbc82e67-1421-4ff2-bd09-426c985addbd-kube-api-access-lvth7\") pod \"leave-v2-kserve-556b4dddf9-wrc2z\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:32.142190 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:32.142050    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:32.445931 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:32.445891    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"]
Sep 16 19:20:32.447429 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:20:32.447401    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc82e67_1421_4ff2_bd09_426c985addbd.slice/crio-6335d5f089f0ede5c03dfc510ec96344588cf2ca13f38944af681b3e0c6a2c87 WatchSource:0}: Error finding container 6335d5f089f0ede5c03dfc510ec96344588cf2ca13f38944af681b3e0c6a2c87: Status 404 returned error can't find the container with id 6335d5f089f0ede5c03dfc510ec96344588cf2ca13f38944af681b3e0c6a2c87
Sep 16 19:20:32.692419 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:32.692383    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z" event={"ID":"dbc82e67-1421-4ff2-bd09-426c985addbd","Type":"ContainerStarted","Data":"e4063dad5207c282fed5d4d16959e3a2baa07c437dfc630d4bf69a183cb6ca53"}
Sep 16 19:20:32.692637 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:32.692441    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z" event={"ID":"dbc82e67-1421-4ff2-bd09-426c985addbd","Type":"ContainerStarted","Data":"6335d5f089f0ede5c03dfc510ec96344588cf2ca13f38944af681b3e0c6a2c87"}
Sep 16 19:20:33.142723 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:33.142686    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:33.211008 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:33.210959    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:33.334733 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:33.334701    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:39.077964 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:39.077891    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:41.146775 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:41.146740    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:41.421854 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:41.421754    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:42.143246 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:42.143209    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:42.156363 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:42.156328    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:42.183483 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:42.183421    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z" podStartSLOduration=13.183400047 podStartE2EDuration="13.183400047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:32.724573049 +0000 UTC m=+2132.527291266" watchObservedRunningTime="2026-09-16 19:20:42.183400047 +0000 UTC m=+2141.986118257"
Sep 16 19:20:42.767384 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:42.767352    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:20:45.751422 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:45.751372    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:45.855376 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:20:45.855345    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:02.428009 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:21:02.427947    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:23.495404 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:21:23.495330    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:35.125513 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:21:35.125478    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:39.706520 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:21:39.706471    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:39.776493 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:21:39.776465    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:45.052615 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:21:45.052564    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:45.156906 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:21:45.156858    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:45.358035 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:21:45.357936    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:45.541921 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:21:45.541885    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"]
Sep 16 19:21:45.629791 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:21:45.629703    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:45.851251 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:21:45.851214    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:50.137858 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:21:50.137819    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:00.563291 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.563228    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z" podUID="dbc82e67-1421-4ff2-bd09-426c985addbd" containerName="main" containerID="cri-o://e4063dad5207c282fed5d4d16959e3a2baa07c437dfc630d4bf69a183cb6ca53" gracePeriod=45
Sep 16 19:22:00.820483 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.820408    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:22:00.910943 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.910899    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-dshm\") pod \"dbc82e67-1421-4ff2-bd09-426c985addbd\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") "
Sep 16 19:22:00.911189 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.910954    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvth7\" (UniqueName: \"kubernetes.io/projected/dbc82e67-1421-4ff2-bd09-426c985addbd-kube-api-access-lvth7\") pod \"dbc82e67-1421-4ff2-bd09-426c985addbd\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") "
Sep 16 19:22:00.911189 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.910992    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-model-cache\") pod \"dbc82e67-1421-4ff2-bd09-426c985addbd\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") "
Sep 16 19:22:00.911189 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.911044    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-home\") pod \"dbc82e67-1421-4ff2-bd09-426c985addbd\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") "
Sep 16 19:22:00.911386 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.911219    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-tmp-dir\") pod \"dbc82e67-1421-4ff2-bd09-426c985addbd\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") "
Sep 16 19:22:00.911386 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.911282    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc82e67-1421-4ff2-bd09-426c985addbd-tls-certs\") pod \"dbc82e67-1421-4ff2-bd09-426c985addbd\" (UID: \"dbc82e67-1421-4ff2-bd09-426c985addbd\") "
Sep 16 19:22:00.911386 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.911339    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-model-cache" (OuterVolumeSpecName: "model-cache") pod "dbc82e67-1421-4ff2-bd09-426c985addbd" (UID: "dbc82e67-1421-4ff2-bd09-426c985addbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:00.911386 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.911353    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-home" (OuterVolumeSpecName: "home") pod "dbc82e67-1421-4ff2-bd09-426c985addbd" (UID: "dbc82e67-1421-4ff2-bd09-426c985addbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:00.911585 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.911449    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dbc82e67-1421-4ff2-bd09-426c985addbd" (UID: "dbc82e67-1421-4ff2-bd09-426c985addbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:00.911644 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.911613    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:22:00.911644 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.911631    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:22:00.911712 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.911651    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:22:00.913557 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.913528    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc82e67-1421-4ff2-bd09-426c985addbd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbc82e67-1421-4ff2-bd09-426c985addbd" (UID: "dbc82e67-1421-4ff2-bd09-426c985addbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:00.913557 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.913542    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc82e67-1421-4ff2-bd09-426c985addbd-kube-api-access-lvth7" (OuterVolumeSpecName: "kube-api-access-lvth7") pod "dbc82e67-1421-4ff2-bd09-426c985addbd" (UID: "dbc82e67-1421-4ff2-bd09-426c985addbd"). InnerVolumeSpecName "kube-api-access-lvth7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:00.913557 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:00.913533    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-dshm" (OuterVolumeSpecName: "dshm") pod "dbc82e67-1421-4ff2-bd09-426c985addbd" (UID: "dbc82e67-1421-4ff2-bd09-426c985addbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:01.012791 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:01.012753    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc82e67-1421-4ff2-bd09-426c985addbd-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:22:01.012791 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:01.012783    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc82e67-1421-4ff2-bd09-426c985addbd-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:22:01.012791 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:01.012792    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvth7\" (UniqueName: \"kubernetes.io/projected/dbc82e67-1421-4ff2-bd09-426c985addbd-kube-api-access-lvth7\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:22:01.209416 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:01.209311    2561 generic.go:356] "Generic (PLEG): container finished" podID="dbc82e67-1421-4ff2-bd09-426c985addbd" containerID="e4063dad5207c282fed5d4d16959e3a2baa07c437dfc630d4bf69a183cb6ca53" exitCode=0
Sep 16 19:22:01.209416 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:01.209359    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"
Sep 16 19:22:01.209416 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:01.209402    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z" event={"ID":"dbc82e67-1421-4ff2-bd09-426c985addbd","Type":"ContainerDied","Data":"e4063dad5207c282fed5d4d16959e3a2baa07c437dfc630d4bf69a183cb6ca53"}
Sep 16 19:22:01.209712 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:01.209463    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z" event={"ID":"dbc82e67-1421-4ff2-bd09-426c985addbd","Type":"ContainerDied","Data":"6335d5f089f0ede5c03dfc510ec96344588cf2ca13f38944af681b3e0c6a2c87"}
Sep 16 19:22:01.209712 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:01.209488    2561 scope.go:117] "RemoveContainer" containerID="e4063dad5207c282fed5d4d16959e3a2baa07c437dfc630d4bf69a183cb6ca53"
Sep 16 19:22:01.219623 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:01.219602    2561 scope.go:117] "RemoveContainer" containerID="e4063dad5207c282fed5d4d16959e3a2baa07c437dfc630d4bf69a183cb6ca53"
Sep 16 19:22:01.219943 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:22:01.219920    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4063dad5207c282fed5d4d16959e3a2baa07c437dfc630d4bf69a183cb6ca53\": container with ID starting with e4063dad5207c282fed5d4d16959e3a2baa07c437dfc630d4bf69a183cb6ca53 not found: ID does not exist" containerID="e4063dad5207c282fed5d4d16959e3a2baa07c437dfc630d4bf69a183cb6ca53"
Sep 16 19:22:01.220027 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:01.219952    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4063dad5207c282fed5d4d16959e3a2baa07c437dfc630d4bf69a183cb6ca53"} err="failed to get container status \"e4063dad5207c282fed5d4d16959e3a2baa07c437dfc630d4bf69a183cb6ca53\": rpc error: code = NotFound desc = could not find container \"e4063dad5207c282fed5d4d16959e3a2baa07c437dfc630d4bf69a183cb6ca53\": container with ID starting with e4063dad5207c282fed5d4d16959e3a2baa07c437dfc630d4bf69a183cb6ca53 not found: ID does not exist"
Sep 16 19:22:01.234770 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:01.234735    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"]
Sep 16 19:22:01.244243 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:01.244216    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z"]
Sep 16 19:22:01.756696 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:01.756646    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wrc2z" podUID="dbc82e67-1421-4ff2-bd09-426c985addbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:22:01.928182 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:01.928146    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:02.736582 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:02.736547    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbc82e67-1421-4ff2-bd09-426c985addbd" path="/var/lib/kubelet/pods/dbc82e67-1421-4ff2-bd09-426c985addbd/volumes"
Sep 16 19:22:06.388411 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:06.388369    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:19.589097 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:19.589033    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:27.564092 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:27.563964    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:28.395346 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:28.395306    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:28.482847 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:28.482816    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:33.978432 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:33.978384    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:34.080443 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.080407    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:34.776149 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.776114    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"]
Sep 16 19:22:34.776573 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.776558    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc82e67-1421-4ff2-bd09-426c985addbd" containerName="main"
Sep 16 19:22:34.776625 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.776577    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc82e67-1421-4ff2-bd09-426c985addbd" containerName="main"
Sep 16 19:22:34.776662 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.776644    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc82e67-1421-4ff2-bd09-426c985addbd" containerName="main"
Sep 16 19:22:34.779673 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.779652    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:34.784139 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.784117    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 16 19:22:34.784273 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.784118    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-clqmt\""
Sep 16 19:22:34.784273 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.784119    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 16 19:22:34.784273 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.784171    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 16 19:22:34.795823 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.795774    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:34.796768 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.796740    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"]
Sep 16 19:22:34.922844 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.922808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:34.923032 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.922851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf87r\" (UniqueName: \"kubernetes.io/projected/a1d9e043-2307-4235-8885-efdceefd32ba-kube-api-access-hf87r\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:34.923032 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.922871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-home\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:34.923032 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.922960    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d9e043-2307-4235-8885-efdceefd32ba-tls-certs\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:34.923032 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.923004    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-model-cache\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:34.923219 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:34.923081    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-dshm\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:35.024015 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.023980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-model-cache\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:35.024538 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.024037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-dshm\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:35.024538 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.024101    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:35.024538 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.024125    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf87r\" (UniqueName: \"kubernetes.io/projected/a1d9e043-2307-4235-8885-efdceefd32ba-kube-api-access-hf87r\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:35.024538 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.024142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-home\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:35.024538 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.024195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d9e043-2307-4235-8885-efdceefd32ba-tls-certs\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:35.024538 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.024488    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-model-cache\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:35.024758 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.024536    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:35.024758 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.024563    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-home\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:35.026570 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.026517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-dshm\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:35.026798 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.026780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d9e043-2307-4235-8885-efdceefd32ba-tls-certs\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:35.034907 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.034880    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf87r\" (UniqueName: \"kubernetes.io/projected/a1d9e043-2307-4235-8885-efdceefd32ba-kube-api-access-hf87r\") pod \"tri-v3-kserve-798c88c66c-mm6xv\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:35.089875 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.089835    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:35.372138 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.372108    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"]
Sep 16 19:22:35.373887 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:22:35.373863    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d9e043_2307_4235_8885_efdceefd32ba.slice/crio-19458acd2515dec7d6b9adaeef560b16950d53dd6a2de53f979c1a4c90ecdbd6 WatchSource:0}: Error finding container 19458acd2515dec7d6b9adaeef560b16950d53dd6a2de53f979c1a4c90ecdbd6: Status 404 returned error can't find the container with id 19458acd2515dec7d6b9adaeef560b16950d53dd6a2de53f979c1a4c90ecdbd6
Sep 16 19:22:35.418995 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:35.418951    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv" event={"ID":"a1d9e043-2307-4235-8885-efdceefd32ba","Type":"ContainerStarted","Data":"19458acd2515dec7d6b9adaeef560b16950d53dd6a2de53f979c1a4c90ecdbd6"}
Sep 16 19:22:36.426315 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:36.426279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv" event={"ID":"a1d9e043-2307-4235-8885-efdceefd32ba","Type":"ContainerStarted","Data":"a9ea46e8e798231ee92327b5cc121e1710462630a506304765842021dd6a6b3d"}
Sep 16 19:22:36.449775 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:36.449715    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv" podStartSLOduration=2.449697522 podStartE2EDuration="2.449697522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:36.448852151 +0000 UTC m=+2256.251570366" watchObservedRunningTime="2026-09-16 19:22:36.449697522 +0000 UTC m=+2256.252415739"
Sep 16 19:22:36.538781 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:36.538735    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:37.090731 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:37.090694    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:42.329583 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:42.329549    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:42.430308 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:42.430261    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:45.090549 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:45.090513    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:45.103392 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:45.103365    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:22:45.486484 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:22:45.486454    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:23:11.699807 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:11.699759    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:12.000439 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.000396    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"]
Sep 16 19:23:12.004699 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.004676    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"
Sep 16 19:23:12.007794 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.007768    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-9n9j2\""
Sep 16 19:23:12.007993 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.007968    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:23:12.008125 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.007968    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:23:12.008539 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.008522    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-j57wq\""
Sep 16 19:23:12.012887 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.012867    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 19:23:12.016350 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.016330    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"]
Sep 16 19:23:12.038262 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.038225    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:12.067157 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.067118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-559zf\" (UniqueName: \"kubernetes.io/projected/06e4c763-a4f6-4c84-a654-f20a8c85a741-kube-api-access-559zf\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt\" (UID: \"06e4c763-a4f6-4c84-a654-f20a8c85a741\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"
Sep 16 19:23:12.067315 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.067173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e4c763-a4f6-4c84-a654-f20a8c85a741-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt\" (UID: \"06e4c763-a4f6-4c84-a654-f20a8c85a741\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"
Sep 16 19:23:12.168039 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.168000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e4c763-a4f6-4c84-a654-f20a8c85a741-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt\" (UID: \"06e4c763-a4f6-4c84-a654-f20a8c85a741\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"
Sep 16 19:23:12.168259 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.168131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-559zf\" (UniqueName: \"kubernetes.io/projected/06e4c763-a4f6-4c84-a654-f20a8c85a741-kube-api-access-559zf\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt\" (UID: \"06e4c763-a4f6-4c84-a654-f20a8c85a741\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"
Sep 16 19:23:12.170736 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.170714    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e4c763-a4f6-4c84-a654-f20a8c85a741-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt\" (UID: \"06e4c763-a4f6-4c84-a654-f20a8c85a741\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"
Sep 16 19:23:12.179586 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.179557    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-559zf\" (UniqueName: \"kubernetes.io/projected/06e4c763-a4f6-4c84-a654-f20a8c85a741-kube-api-access-559zf\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt\" (UID: \"06e4c763-a4f6-4c84-a654-f20a8c85a741\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"
Sep 16 19:23:12.316003 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.315902    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"
Sep 16 19:23:12.602285 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.602250    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"]
Sep 16 19:23:12.603504 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:23:12.603477    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e4c763_a4f6_4c84_a654_f20a8c85a741.slice/crio-ab0587dc7b8852537dd0569f13970b1815b48cba91d9af96bd25281d7b34f1e5 WatchSource:0}: Error finding container ab0587dc7b8852537dd0569f13970b1815b48cba91d9af96bd25281d7b34f1e5: Status 404 returned error can't find the container with id ab0587dc7b8852537dd0569f13970b1815b48cba91d9af96bd25281d7b34f1e5
Sep 16 19:23:12.631540 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:12.631508    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt" event={"ID":"06e4c763-a4f6-4c84-a654-f20a8c85a741","Type":"ContainerStarted","Data":"ab0587dc7b8852537dd0569f13970b1815b48cba91d9af96bd25281d7b34f1e5"}
Sep 16 19:23:13.639628 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:13.639591    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt" event={"ID":"06e4c763-a4f6-4c84-a654-f20a8c85a741","Type":"ContainerStarted","Data":"55a2ca02781e3c9039ed64a3f86dc797398327a47d625693fd82bfaaf1def89b"}
Sep 16 19:23:13.640123 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:13.639660    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"
Sep 16 19:23:13.690101 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:13.690025    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt" podStartSLOduration=2.690007567 podStartE2EDuration="2.690007567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:13.688687493 +0000 UTC m=+2293.491405738" watchObservedRunningTime="2026-09-16 19:23:13.690007567 +0000 UTC m=+2293.492725783"
Sep 16 19:23:18.887240 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:18.887196    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:25.947194 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:25.947159    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:26.014117 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:26.014048    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:44.647609 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:23:44.647575    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"
Sep 16 19:24:10.308807 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:10.308772    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:10.408383 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:10.408333    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:10.510324 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:10.510291    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:15.303927 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:15.303894    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"]
Sep 16 19:24:15.355888 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:15.355850    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:26.602164 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:26.602127    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:26.912141 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:26.912028    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:30.298508 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.298473    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:30.324691 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.324642    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv" podUID="a1d9e043-2307-4235-8885-efdceefd32ba" containerName="main" containerID="cri-o://a9ea46e8e798231ee92327b5cc121e1710462630a506304765842021dd6a6b3d" gracePeriod=45
Sep 16 19:24:30.475979 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.475936    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv" podUID="a1d9e043-2307-4235-8885-efdceefd32ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 16 19:24:30.583439 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.583409    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:24:30.615945 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.615895    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-tmp-dir\") pod \"a1d9e043-2307-4235-8885-efdceefd32ba\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") "
Sep 16 19:24:30.616150 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.615985    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hf87r\" (UniqueName: \"kubernetes.io/projected/a1d9e043-2307-4235-8885-efdceefd32ba-kube-api-access-hf87r\") pod \"a1d9e043-2307-4235-8885-efdceefd32ba\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") "
Sep 16 19:24:30.616150 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.616021    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d9e043-2307-4235-8885-efdceefd32ba-tls-certs\") pod \"a1d9e043-2307-4235-8885-efdceefd32ba\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") "
Sep 16 19:24:30.616150 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.616103    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-home\") pod \"a1d9e043-2307-4235-8885-efdceefd32ba\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") "
Sep 16 19:24:30.616343 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.616166    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-dshm\") pod \"a1d9e043-2307-4235-8885-efdceefd32ba\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") "
Sep 16 19:24:30.616343 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.616188    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a1d9e043-2307-4235-8885-efdceefd32ba" (UID: "a1d9e043-2307-4235-8885-efdceefd32ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:30.616343 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.616202    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-model-cache\") pod \"a1d9e043-2307-4235-8885-efdceefd32ba\" (UID: \"a1d9e043-2307-4235-8885-efdceefd32ba\") "
Sep 16 19:24:30.616530 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.616506    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.616806 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.616778    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1d9e043-2307-4235-8885-efdceefd32ba" (UID: "a1d9e043-2307-4235-8885-efdceefd32ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:30.617499 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.616956    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-home" (OuterVolumeSpecName: "home") pod "a1d9e043-2307-4235-8885-efdceefd32ba" (UID: "a1d9e043-2307-4235-8885-efdceefd32ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:30.619345 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.619311    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1d9e043-2307-4235-8885-efdceefd32ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1d9e043-2307-4235-8885-efdceefd32ba" (UID: "a1d9e043-2307-4235-8885-efdceefd32ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:30.619433 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.619365    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d9e043-2307-4235-8885-efdceefd32ba-kube-api-access-hf87r" (OuterVolumeSpecName: "kube-api-access-hf87r") pod "a1d9e043-2307-4235-8885-efdceefd32ba" (UID: "a1d9e043-2307-4235-8885-efdceefd32ba"). InnerVolumeSpecName "kube-api-access-hf87r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:30.619433 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.619376    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-dshm" (OuterVolumeSpecName: "dshm") pod "a1d9e043-2307-4235-8885-efdceefd32ba" (UID: "a1d9e043-2307-4235-8885-efdceefd32ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:30.717183 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.717143    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf87r\" (UniqueName: \"kubernetes.io/projected/a1d9e043-2307-4235-8885-efdceefd32ba-kube-api-access-hf87r\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.717183 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.717176    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d9e043-2307-4235-8885-efdceefd32ba-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.717183 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.717185    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.717183 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.717193    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.717460 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:30.717201    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d9e043-2307-4235-8885-efdceefd32ba-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:24:31.091725 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:31.091697    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"
Sep 16 19:24:31.091919 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:31.091732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv" event={"ID":"a1d9e043-2307-4235-8885-efdceefd32ba","Type":"ContainerDied","Data":"a9ea46e8e798231ee92327b5cc121e1710462630a506304765842021dd6a6b3d"}
Sep 16 19:24:31.091919 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:31.091656    2561 generic.go:356] "Generic (PLEG): container finished" podID="a1d9e043-2307-4235-8885-efdceefd32ba" containerID="a9ea46e8e798231ee92327b5cc121e1710462630a506304765842021dd6a6b3d" exitCode=0
Sep 16 19:24:31.091919 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:31.091792    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv" event={"ID":"a1d9e043-2307-4235-8885-efdceefd32ba","Type":"ContainerDied","Data":"19458acd2515dec7d6b9adaeef560b16950d53dd6a2de53f979c1a4c90ecdbd6"}
Sep 16 19:24:31.091919 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:31.091816    2561 scope.go:117] "RemoveContainer" containerID="a9ea46e8e798231ee92327b5cc121e1710462630a506304765842021dd6a6b3d"
Sep 16 19:24:31.102318 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:31.101954    2561 scope.go:117] "RemoveContainer" containerID="a9ea46e8e798231ee92327b5cc121e1710462630a506304765842021dd6a6b3d"
Sep 16 19:24:31.102318 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:24:31.102300    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ea46e8e798231ee92327b5cc121e1710462630a506304765842021dd6a6b3d\": container with ID starting with a9ea46e8e798231ee92327b5cc121e1710462630a506304765842021dd6a6b3d not found: ID does not exist" containerID="a9ea46e8e798231ee92327b5cc121e1710462630a506304765842021dd6a6b3d"
Sep 16 19:24:31.102505 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:31.102330    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ea46e8e798231ee92327b5cc121e1710462630a506304765842021dd6a6b3d"} err="failed to get container status \"a9ea46e8e798231ee92327b5cc121e1710462630a506304765842021dd6a6b3d\": rpc error: code = NotFound desc = could not find container \"a9ea46e8e798231ee92327b5cc121e1710462630a506304765842021dd6a6b3d\": container with ID starting with a9ea46e8e798231ee92327b5cc121e1710462630a506304765842021dd6a6b3d not found: ID does not exist"
Sep 16 19:24:31.122369 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:31.122334    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"]
Sep 16 19:24:31.140848 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:31.140818    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mm6xv"]
Sep 16 19:24:31.163949 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:31.163920    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:32.174185 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:32.174139    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:32.734785 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:32.734743    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1d9e043-2307-4235-8885-efdceefd32ba" path="/var/lib/kubelet/pods/a1d9e043-2307-4235-8885-efdceefd32ba/volumes"
Sep 16 19:24:45.262774 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:45.262732    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:51.770844 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.770800    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"]
Sep 16 19:24:51.771445 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.771423    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d9e043-2307-4235-8885-efdceefd32ba" containerName="main"
Sep 16 19:24:51.771524 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.771449    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d9e043-2307-4235-8885-efdceefd32ba" containerName="main"
Sep 16 19:24:51.771586 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.771576    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1d9e043-2307-4235-8885-efdceefd32ba" containerName="main"
Sep 16 19:24:51.774628 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.774605    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:51.777673 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.777649    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-h7hbv\""
Sep 16 19:24:51.778730 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.778700    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 16 19:24:51.778931 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.778767    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 16 19:24:51.779049 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.778814    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 16 19:24:51.793372 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.793341    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"]
Sep 16 19:24:51.815366 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.815335    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:51.908934 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.908890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3925ce84-45f4-49fa-badb-a28936422772-tls-certs\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:51.909170 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.909005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t86p\" (UniqueName: \"kubernetes.io/projected/3925ce84-45f4-49fa-badb-a28936422772-kube-api-access-2t86p\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:51.909170 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.909028    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-model-cache\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:51.909170 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.909112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-dshm\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:51.909170 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.909137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-home\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:51.909351 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:51.909207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:52.010562 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:52.010525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t86p\" (UniqueName: \"kubernetes.io/projected/3925ce84-45f4-49fa-badb-a28936422772-kube-api-access-2t86p\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:52.010562 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:52.010570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-model-cache\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:52.010862 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:52.010585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-dshm\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:52.010862 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:52.010737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-home\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:52.010862 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:52.010790    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:52.011038 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:52.010879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3925ce84-45f4-49fa-badb-a28936422772-tls-certs\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:52.011145 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:52.011029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-model-cache\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:52.011208 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:52.011174    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-home\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:52.011308 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:52.011283    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:52.013232 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:52.013204    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-dshm\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:52.013569 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:52.013551    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3925ce84-45f4-49fa-badb-a28936422772-tls-certs\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:52.021499 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:52.021415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t86p\" (UniqueName: \"kubernetes.io/projected/3925ce84-45f4-49fa-badb-a28936422772-kube-api-access-2t86p\") pod \"late-v1-kserve-56cbc8456c-6m8ml\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:52.086023 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:52.085984    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:52.367515 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:52.367488    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"]
Sep 16 19:24:52.369743 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:24:52.369715    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3925ce84_45f4_49fa_badb_a28936422772.slice/crio-fc929dfe7e5443947be33fdea3eba13485b28f24d7033fc04afe8fa16bf1500a WatchSource:0}: Error finding container fc929dfe7e5443947be33fdea3eba13485b28f24d7033fc04afe8fa16bf1500a: Status 404 returned error can't find the container with id fc929dfe7e5443947be33fdea3eba13485b28f24d7033fc04afe8fa16bf1500a
Sep 16 19:24:53.219107 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:53.219071    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml" event={"ID":"3925ce84-45f4-49fa-badb-a28936422772","Type":"ContainerStarted","Data":"7952fcba39a98c257e2b9187f9ba4c0e74f15cd29b0d8a10a85b37d239bea360"}
Sep 16 19:24:53.219506 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:53.219134    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml" event={"ID":"3925ce84-45f4-49fa-badb-a28936422772","Type":"ContainerStarted","Data":"fc929dfe7e5443947be33fdea3eba13485b28f24d7033fc04afe8fa16bf1500a"}
Sep 16 19:24:53.240464 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:53.240403    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml" podStartSLOduration=2.240385697 podStartE2EDuration="2.240385697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:53.238967176 +0000 UTC m=+2393.041685393" watchObservedRunningTime="2026-09-16 19:24:53.240385697 +0000 UTC m=+2393.043103950"
Sep 16 19:24:53.762259 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:53.762223    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:54.086668 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:54.086575    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:24:54.235833 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:24:54.235760    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:00.464336 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:25:00.464295    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:00.536199 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:25:00.536165    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:00.938939 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:25:00.938849    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:25:00.944995 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:25:00.944969    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:25:02.086873 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:25:02.086824    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:25:02.099792 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:25:02.099767    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:25:02.279996 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:25:02.279965    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:25:02.713485 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:25:02.713446    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:02.812010 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:25:02.811978    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:25.093584 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:25:25.093503    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:52.784642 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:25:52.784601    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"]
Sep 16 19:25:52.790707 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:25:52.790671    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:52.874697 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:25:52.874660    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:54.647121 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:25:54.647052    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt" podUID="06e4c763-a4f6-4c84-a654-f20a8c85a741" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:04.647301 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:04.647261    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt" podUID="06e4c763-a4f6-4c84-a654-f20a8c85a741" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:06.260218 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:06.260175    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:06.394464 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:06.394434    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:06.494354 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:06.494318    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:06.756000 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:06.755964    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"]
Sep 16 19:26:06.784779 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:06.784745    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:07.805599 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:07.805512    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt" podUID="06e4c763-a4f6-4c84-a654-f20a8c85a741" containerName="main" containerID="cri-o://55a2ca02781e3c9039ed64a3f86dc797398327a47d625693fd82bfaaf1def89b" gracePeriod=45
Sep 16 19:26:11.319537 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:11.319504    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:13.488248 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:13.488211    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:14.647457 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:14.647416    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt" podUID="06e4c763-a4f6-4c84-a654-f20a8c85a741" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:14.647878 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:14.647549    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"
Sep 16 19:26:21.779142 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:21.779070    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml" podUID="3925ce84-45f4-49fa-badb-a28936422772" containerName="main" containerID="cri-o://7952fcba39a98c257e2b9187f9ba4c0e74f15cd29b0d8a10a85b37d239bea360" gracePeriod=45
Sep 16 19:26:22.034187 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.034119    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:26:22.168128 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.168090    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t86p\" (UniqueName: \"kubernetes.io/projected/3925ce84-45f4-49fa-badb-a28936422772-kube-api-access-2t86p\") pod \"3925ce84-45f4-49fa-badb-a28936422772\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") "
Sep 16 19:26:22.168377 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.168178    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-home\") pod \"3925ce84-45f4-49fa-badb-a28936422772\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") "
Sep 16 19:26:22.168377 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.168258    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-dshm\") pod \"3925ce84-45f4-49fa-badb-a28936422772\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") "
Sep 16 19:26:22.168377 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.168303    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-tmp-dir\") pod \"3925ce84-45f4-49fa-badb-a28936422772\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") "
Sep 16 19:26:22.168377 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.168326    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-model-cache\") pod \"3925ce84-45f4-49fa-badb-a28936422772\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") "
Sep 16 19:26:22.168377 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.168368    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3925ce84-45f4-49fa-badb-a28936422772-tls-certs\") pod \"3925ce84-45f4-49fa-badb-a28936422772\" (UID: \"3925ce84-45f4-49fa-badb-a28936422772\") "
Sep 16 19:26:22.168660 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.168504    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-home" (OuterVolumeSpecName: "home") pod "3925ce84-45f4-49fa-badb-a28936422772" (UID: "3925ce84-45f4-49fa-badb-a28936422772"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:22.168660 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.168557    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3925ce84-45f4-49fa-badb-a28936422772" (UID: "3925ce84-45f4-49fa-badb-a28936422772"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:22.168660 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.168600    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-model-cache" (OuterVolumeSpecName: "model-cache") pod "3925ce84-45f4-49fa-badb-a28936422772" (UID: "3925ce84-45f4-49fa-badb-a28936422772"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:22.168769 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.168679    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:26:22.168769 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.168696    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:26:22.168769 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.168706    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:26:22.170747 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.170721    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3925ce84-45f4-49fa-badb-a28936422772-kube-api-access-2t86p" (OuterVolumeSpecName: "kube-api-access-2t86p") pod "3925ce84-45f4-49fa-badb-a28936422772" (UID: "3925ce84-45f4-49fa-badb-a28936422772"). InnerVolumeSpecName "kube-api-access-2t86p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:22.170851 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.170758    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-dshm" (OuterVolumeSpecName: "dshm") pod "3925ce84-45f4-49fa-badb-a28936422772" (UID: "3925ce84-45f4-49fa-badb-a28936422772"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:22.170851 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.170749    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3925ce84-45f4-49fa-badb-a28936422772-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3925ce84-45f4-49fa-badb-a28936422772" (UID: "3925ce84-45f4-49fa-badb-a28936422772"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:22.269466 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.269426    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3925ce84-45f4-49fa-badb-a28936422772-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:26:22.269466 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.269459    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3925ce84-45f4-49fa-badb-a28936422772-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:26:22.269466 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.269471    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t86p\" (UniqueName: \"kubernetes.io/projected/3925ce84-45f4-49fa-badb-a28936422772-kube-api-access-2t86p\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:26:22.728824 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.728793    2561 generic.go:356] "Generic (PLEG): container finished" podID="3925ce84-45f4-49fa-badb-a28936422772" containerID="7952fcba39a98c257e2b9187f9ba4c0e74f15cd29b0d8a10a85b37d239bea360" exitCode=0
Sep 16 19:26:22.729010 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.728877    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"
Sep 16 19:26:22.734227 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.734198    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml" event={"ID":"3925ce84-45f4-49fa-badb-a28936422772","Type":"ContainerDied","Data":"7952fcba39a98c257e2b9187f9ba4c0e74f15cd29b0d8a10a85b37d239bea360"}
Sep 16 19:26:22.734398 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.734252    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml" event={"ID":"3925ce84-45f4-49fa-badb-a28936422772","Type":"ContainerDied","Data":"fc929dfe7e5443947be33fdea3eba13485b28f24d7033fc04afe8fa16bf1500a"}
Sep 16 19:26:22.734398 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.734282    2561 scope.go:117] "RemoveContainer" containerID="7952fcba39a98c257e2b9187f9ba4c0e74f15cd29b0d8a10a85b37d239bea360"
Sep 16 19:26:22.743747 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.743727    2561 scope.go:117] "RemoveContainer" containerID="7952fcba39a98c257e2b9187f9ba4c0e74f15cd29b0d8a10a85b37d239bea360"
Sep 16 19:26:22.744044 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:26:22.744026    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7952fcba39a98c257e2b9187f9ba4c0e74f15cd29b0d8a10a85b37d239bea360\": container with ID starting with 7952fcba39a98c257e2b9187f9ba4c0e74f15cd29b0d8a10a85b37d239bea360 not found: ID does not exist" containerID="7952fcba39a98c257e2b9187f9ba4c0e74f15cd29b0d8a10a85b37d239bea360"
Sep 16 19:26:22.744150 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.744049    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7952fcba39a98c257e2b9187f9ba4c0e74f15cd29b0d8a10a85b37d239bea360"} err="failed to get container status \"7952fcba39a98c257e2b9187f9ba4c0e74f15cd29b0d8a10a85b37d239bea360\": rpc error: code = NotFound desc = could not find container \"7952fcba39a98c257e2b9187f9ba4c0e74f15cd29b0d8a10a85b37d239bea360\": container with ID starting with 7952fcba39a98c257e2b9187f9ba4c0e74f15cd29b0d8a10a85b37d239bea360 not found: ID does not exist"
Sep 16 19:26:22.754881 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.754854    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"]
Sep 16 19:26:22.765755 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:22.765723    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-6m8ml"]
Sep 16 19:26:24.646917 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:24.646879    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt" podUID="06e4c763-a4f6-4c84-a654-f20a8c85a741" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:24.734467 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:24.734433    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3925ce84-45f4-49fa-badb-a28936422772" path="/var/lib/kubelet/pods/3925ce84-45f4-49fa-badb-a28936422772/volumes"
Sep 16 19:26:34.647757 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:34.647714    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt" podUID="06e4c763-a4f6-4c84-a654-f20a8c85a741" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:35.202951 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:35.202915    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:38.067884 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.067817    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"
Sep 16 19:26:38.117016 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.116985    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e4c763-a4f6-4c84-a654-f20a8c85a741-tls-certs\") pod \"06e4c763-a4f6-4c84-a654-f20a8c85a741\" (UID: \"06e4c763-a4f6-4c84-a654-f20a8c85a741\") "
Sep 16 19:26:38.117189 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.117094    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-559zf\" (UniqueName: \"kubernetes.io/projected/06e4c763-a4f6-4c84-a654-f20a8c85a741-kube-api-access-559zf\") pod \"06e4c763-a4f6-4c84-a654-f20a8c85a741\" (UID: \"06e4c763-a4f6-4c84-a654-f20a8c85a741\") "
Sep 16 19:26:38.119529 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.119488    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e4c763-a4f6-4c84-a654-f20a8c85a741-kube-api-access-559zf" (OuterVolumeSpecName: "kube-api-access-559zf") pod "06e4c763-a4f6-4c84-a654-f20a8c85a741" (UID: "06e4c763-a4f6-4c84-a654-f20a8c85a741"). InnerVolumeSpecName "kube-api-access-559zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:38.119677 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.119534    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e4c763-a4f6-4c84-a654-f20a8c85a741-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06e4c763-a4f6-4c84-a654-f20a8c85a741" (UID: "06e4c763-a4f6-4c84-a654-f20a8c85a741"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:38.217792 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.217757    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-559zf\" (UniqueName: \"kubernetes.io/projected/06e4c763-a4f6-4c84-a654-f20a8c85a741-kube-api-access-559zf\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:26:38.217792 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.217787    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e4c763-a4f6-4c84-a654-f20a8c85a741-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:26:38.819194 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.819157    2561 generic.go:356] "Generic (PLEG): container finished" podID="06e4c763-a4f6-4c84-a654-f20a8c85a741" containerID="55a2ca02781e3c9039ed64a3f86dc797398327a47d625693fd82bfaaf1def89b" exitCode=0
Sep 16 19:26:38.819194 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.819198    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt" event={"ID":"06e4c763-a4f6-4c84-a654-f20a8c85a741","Type":"ContainerDied","Data":"55a2ca02781e3c9039ed64a3f86dc797398327a47d625693fd82bfaaf1def89b"}
Sep 16 19:26:38.819456 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.819205    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"
Sep 16 19:26:38.819456 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.819243    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt" event={"ID":"06e4c763-a4f6-4c84-a654-f20a8c85a741","Type":"ContainerDied","Data":"ab0587dc7b8852537dd0569f13970b1815b48cba91d9af96bd25281d7b34f1e5"}
Sep 16 19:26:38.819456 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.819266    2561 scope.go:117] "RemoveContainer" containerID="55a2ca02781e3c9039ed64a3f86dc797398327a47d625693fd82bfaaf1def89b"
Sep 16 19:26:38.829031 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.829011    2561 scope.go:117] "RemoveContainer" containerID="55a2ca02781e3c9039ed64a3f86dc797398327a47d625693fd82bfaaf1def89b"
Sep 16 19:26:38.829374 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:26:38.829355    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a2ca02781e3c9039ed64a3f86dc797398327a47d625693fd82bfaaf1def89b\": container with ID starting with 55a2ca02781e3c9039ed64a3f86dc797398327a47d625693fd82bfaaf1def89b not found: ID does not exist" containerID="55a2ca02781e3c9039ed64a3f86dc797398327a47d625693fd82bfaaf1def89b"
Sep 16 19:26:38.829432 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.829383    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a2ca02781e3c9039ed64a3f86dc797398327a47d625693fd82bfaaf1def89b"} err="failed to get container status \"55a2ca02781e3c9039ed64a3f86dc797398327a47d625693fd82bfaaf1def89b\": rpc error: code = NotFound desc = could not find container \"55a2ca02781e3c9039ed64a3f86dc797398327a47d625693fd82bfaaf1def89b\": container with ID starting with 55a2ca02781e3c9039ed64a3f86dc797398327a47d625693fd82bfaaf1def89b not found: ID does not exist"
Sep 16 19:26:38.838137 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.838102    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"]
Sep 16 19:26:38.840617 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:38.840593    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef6qlt"]
Sep 16 19:26:40.734708 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:40.734662    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06e4c763-a4f6-4c84-a654-f20a8c85a741" path="/var/lib/kubelet/pods/06e4c763-a4f6-4c84-a654-f20a8c85a741/volumes"
Sep 16 19:26:40.958376 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:40.958346    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:42.906535 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:42.906498    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:43.003929 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:43.003896    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:47.411025 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:47.410992    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:47.503478 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:47.503447    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:49.942026 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:49.941962    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:51.853493 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:51.853386    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:52.151442 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.151354    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"]
Sep 16 19:26:52.151784 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.151771    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3925ce84-45f4-49fa-badb-a28936422772" containerName="main"
Sep 16 19:26:52.151835 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.151785    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3925ce84-45f4-49fa-badb-a28936422772" containerName="main"
Sep 16 19:26:52.151835 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.151795    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06e4c763-a4f6-4c84-a654-f20a8c85a741" containerName="main"
Sep 16 19:26:52.151835 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.151800    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="06e4c763-a4f6-4c84-a654-f20a8c85a741" containerName="main"
Sep 16 19:26:52.151934 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.151867    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="06e4c763-a4f6-4c84-a654-f20a8c85a741" containerName="main"
Sep 16 19:26:52.151934 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.151877    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3925ce84-45f4-49fa-badb-a28936422772" containerName="main"
Sep 16 19:26:52.156135 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.156119    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"
Sep 16 19:26:52.166792 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.166769    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:26:52.172830 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.172764    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:26:52.173031 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.173008    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-gbshk\""
Sep 16 19:26:52.173231 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.173008    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-f9d2g\""
Sep 16 19:26:52.173231 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.173016    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 19:26:52.183431 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.183406    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"]
Sep 16 19:26:52.243681 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.243633    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdc7d\" (UniqueName: \"kubernetes.io/projected/051c9cb7-e2ef-4518-9111-c860a9c8f1ae-kube-api-access-rdc7d\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5\" (UID: \"051c9cb7-e2ef-4518-9111-c860a9c8f1ae\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"
Sep 16 19:26:52.243897 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.243860    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/051c9cb7-e2ef-4518-9111-c860a9c8f1ae-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5\" (UID: \"051c9cb7-e2ef-4518-9111-c860a9c8f1ae\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"
Sep 16 19:26:52.251338 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.251305    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:52.344757 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.344717    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/051c9cb7-e2ef-4518-9111-c860a9c8f1ae-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5\" (UID: \"051c9cb7-e2ef-4518-9111-c860a9c8f1ae\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"
Sep 16 19:26:52.344941 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.344782    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdc7d\" (UniqueName: \"kubernetes.io/projected/051c9cb7-e2ef-4518-9111-c860a9c8f1ae-kube-api-access-rdc7d\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5\" (UID: \"051c9cb7-e2ef-4518-9111-c860a9c8f1ae\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"
Sep 16 19:26:52.347496 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.347466    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/051c9cb7-e2ef-4518-9111-c860a9c8f1ae-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5\" (UID: \"051c9cb7-e2ef-4518-9111-c860a9c8f1ae\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"
Sep 16 19:26:52.366273 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.366246    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdc7d\" (UniqueName: \"kubernetes.io/projected/051c9cb7-e2ef-4518-9111-c860a9c8f1ae-kube-api-access-rdc7d\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5\" (UID: \"051c9cb7-e2ef-4518-9111-c860a9c8f1ae\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"
Sep 16 19:26:52.466651 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.466617    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"
Sep 16 19:26:52.753722 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.753695    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"]
Sep 16 19:26:52.755536 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:26:52.755504    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051c9cb7_e2ef_4518_9111_c860a9c8f1ae.slice/crio-e9b7d4d66741bb06b457d99f674917560ba88991caff7b04d21b4ae1de93f96c WatchSource:0}: Error finding container e9b7d4d66741bb06b457d99f674917560ba88991caff7b04d21b4ae1de93f96c: Status 404 returned error can't find the container with id e9b7d4d66741bb06b457d99f674917560ba88991caff7b04d21b4ae1de93f96c
Sep 16 19:26:52.904396 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.904356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5" event={"ID":"051c9cb7-e2ef-4518-9111-c860a9c8f1ae","Type":"ContainerStarted","Data":"4071390aee70580c3e2eeb6ce1c691db7fae43baebc6460941db0dce84bab18c"}
Sep 16 19:26:52.904848 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.904423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5" event={"ID":"051c9cb7-e2ef-4518-9111-c860a9c8f1ae","Type":"ContainerStarted","Data":"e9b7d4d66741bb06b457d99f674917560ba88991caff7b04d21b4ae1de93f96c"}
Sep 16 19:26:52.904848 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.904548    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"
Sep 16 19:26:52.931346 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:52.931279    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5" podStartSLOduration=0.931255796 podStartE2EDuration="931.255796ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:52.930587837 +0000 UTC m=+2512.733306067" watchObservedRunningTime="2026-09-16 19:26:52.931255796 +0000 UTC m=+2512.733974013"
Sep 16 19:26:55.894644 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:55.894607    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:55.996424 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:26:55.996388    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:07.062633 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:27:07.062593    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:09.365208 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:27:09.365164    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:23.912306 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:27:23.912273    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"
Sep 16 19:27:26.249213 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:27:26.249183    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:26.351506 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:27:26.351472    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:26.455955 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:27:26.455921    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:30.403346 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:27:30.403312    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:30.648352 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:27:30.648313    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:31.356080 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:27:31.356014    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:36.927531 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:27:36.927498    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:40.206577 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:27:40.206546    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:09.717391 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:09.717355    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:15.596004 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:15.595965    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:15.698923 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:15.698887    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:18.342657 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.342616    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:18.443523 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.443475    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:18.857467 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.857430    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"]
Sep 16 19:28:18.861248 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.861225    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:18.864169 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.864137    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 16 19:28:18.865356 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.865337    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 16 19:28:18.865452 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.865346    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-rf6cs\""
Sep 16 19:28:18.865452 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.865406    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Sep 16 19:28:18.871472 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.871447    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"]
Sep 16 19:28:18.875296 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.875272    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:18.942178 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.942141    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:18.942381 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.942188    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhrkl\" (UniqueName: \"kubernetes.io/projected/3e5704f0-871a-44aa-a49b-3d76203b385e-kube-api-access-dhrkl\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:18.942381 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.942260    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-home\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:18.942381 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.942281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:18.942381 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.942307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5704f0-871a-44aa-a49b-3d76203b385e-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:18.942381 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:18.942328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-dshm\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:19.043084 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:19.043030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-home\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:19.043084 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:19.043089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:19.043355 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:19.043123    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5704f0-871a-44aa-a49b-3d76203b385e-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:19.043355 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:19.043148    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-dshm\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:19.043355 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:19.043198    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:19.043355 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:19.043235    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhrkl\" (UniqueName: \"kubernetes.io/projected/3e5704f0-871a-44aa-a49b-3d76203b385e-kube-api-access-dhrkl\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:19.043573 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:19.043508    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-home\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:19.043573 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:19.043546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:19.044166 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:19.043996    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:19.045830 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:19.045811    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-dshm\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:19.046102 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:19.046086    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5704f0-871a-44aa-a49b-3d76203b385e-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:19.052743 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:19.052718    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhrkl\" (UniqueName: \"kubernetes.io/projected/3e5704f0-871a-44aa-a49b-3d76203b385e-kube-api-access-dhrkl\") pod \"roll-v1-kserve-69cf8bcfc-zhdwt\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:19.173434 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:19.173335    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:19.457736 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:19.457636    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"]
Sep 16 19:28:19.462887 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:28:19.462858    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5704f0_871a_44aa_a49b_3d76203b385e.slice/crio-d8d505ffa2bd83ce289e95152ed3cd6f5fb2f70ceddd5fd5d455d0de9614d48c WatchSource:0}: Error finding container d8d505ffa2bd83ce289e95152ed3cd6f5fb2f70ceddd5fd5d455d0de9614d48c: Status 404 returned error can't find the container with id d8d505ffa2bd83ce289e95152ed3cd6f5fb2f70ceddd5fd5d455d0de9614d48c
Sep 16 19:28:20.411727 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:20.411682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt" event={"ID":"3e5704f0-871a-44aa-a49b-3d76203b385e","Type":"ContainerStarted","Data":"90ee17f8bdccaa3e62255185dad79d136d128ddd8ded46398fe55132a45f4995"}
Sep 16 19:28:20.411972 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:20.411758    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt" event={"ID":"3e5704f0-871a-44aa-a49b-3d76203b385e","Type":"ContainerStarted","Data":"d8d505ffa2bd83ce289e95152ed3cd6f5fb2f70ceddd5fd5d455d0de9614d48c"}
Sep 16 19:28:20.435929 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:20.435870    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt" podStartSLOduration=2.435848285 podStartE2EDuration="2.435848285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:20.435613368 +0000 UTC m=+2600.238331584" watchObservedRunningTime="2026-09-16 19:28:20.435848285 +0000 UTC m=+2600.238566501"
Sep 16 19:28:21.174199 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:21.174106    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:23.344372 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:23.344341    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:29.173832 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:29.173789    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:29.186915 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:29.186887    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:29.485164 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:29.485131    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:28:32.611586 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:32.611550    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:38.538507 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:38.538442    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:38.645407 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:38.645373    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:55.863944 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:55.863879    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:55.960800 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:28:55.960753    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:42.390278 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:29:42.390238    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"]
Sep 16 19:29:42.392313 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:29:42.392284    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:42.493760 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:29:42.493723    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:43.912086 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:29:43.912022    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5" podUID="051c9cb7-e2ef-4518-9111-c860a9c8f1ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:53.911795 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:29:53.911753    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5" podUID="051c9cb7-e2ef-4518-9111-c860a9c8f1ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:57.412687 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:29:57.412640    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5" podUID="051c9cb7-e2ef-4518-9111-c860a9c8f1ae" containerName="main" containerID="cri-o://4071390aee70580c3e2eeb6ce1c691db7fae43baebc6460941db0dce84bab18c" gracePeriod=45
Sep 16 19:30:00.982996 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:00.982968    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:30:00.992133 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:00.992107    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:30:03.911455 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:03.911412    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5" podUID="051c9cb7-e2ef-4518-9111-c860a9c8f1ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:30:03.911933 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:03.911549    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"
Sep 16 19:30:13.911736 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:13.911679    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5" podUID="051c9cb7-e2ef-4518-9111-c860a9c8f1ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:30:16.875960 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:16.875918    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:16.987174 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:16.987139    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:17.123613 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:17.123580    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:17.524829 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:17.524783    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"]
Sep 16 19:30:17.533569 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:17.533538    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:21.937885 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:21.937852    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:23.912109 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:23.912051    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5" podUID="051c9cb7-e2ef-4518-9111-c860a9c8f1ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:30:27.685853 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:27.685770    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"
Sep 16 19:30:27.740402 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:27.740359    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdc7d\" (UniqueName: \"kubernetes.io/projected/051c9cb7-e2ef-4518-9111-c860a9c8f1ae-kube-api-access-rdc7d\") pod \"051c9cb7-e2ef-4518-9111-c860a9c8f1ae\" (UID: \"051c9cb7-e2ef-4518-9111-c860a9c8f1ae\") "
Sep 16 19:30:27.740614 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:27.740464    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/051c9cb7-e2ef-4518-9111-c860a9c8f1ae-tls-certs\") pod \"051c9cb7-e2ef-4518-9111-c860a9c8f1ae\" (UID: \"051c9cb7-e2ef-4518-9111-c860a9c8f1ae\") "
Sep 16 19:30:27.742964 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:27.742928    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051c9cb7-e2ef-4518-9111-c860a9c8f1ae-kube-api-access-rdc7d" (OuterVolumeSpecName: "kube-api-access-rdc7d") pod "051c9cb7-e2ef-4518-9111-c860a9c8f1ae" (UID: "051c9cb7-e2ef-4518-9111-c860a9c8f1ae"). InnerVolumeSpecName "kube-api-access-rdc7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:27.743107 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:27.742970    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051c9cb7-e2ef-4518-9111-c860a9c8f1ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "051c9cb7-e2ef-4518-9111-c860a9c8f1ae" (UID: "051c9cb7-e2ef-4518-9111-c860a9c8f1ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:27.841783 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:27.841743    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdc7d\" (UniqueName: \"kubernetes.io/projected/051c9cb7-e2ef-4518-9111-c860a9c8f1ae-kube-api-access-rdc7d\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:30:27.841783 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:27.841773    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/051c9cb7-e2ef-4518-9111-c860a9c8f1ae-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:30:28.142257 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:28.142222    2561 generic.go:356] "Generic (PLEG): container finished" podID="051c9cb7-e2ef-4518-9111-c860a9c8f1ae" containerID="4071390aee70580c3e2eeb6ce1c691db7fae43baebc6460941db0dce84bab18c" exitCode=0
Sep 16 19:30:28.142432 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:28.142270    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"
Sep 16 19:30:28.142432 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:28.142307    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5" event={"ID":"051c9cb7-e2ef-4518-9111-c860a9c8f1ae","Type":"ContainerDied","Data":"4071390aee70580c3e2eeb6ce1c691db7fae43baebc6460941db0dce84bab18c"}
Sep 16 19:30:28.142432 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:28.142364    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5" event={"ID":"051c9cb7-e2ef-4518-9111-c860a9c8f1ae","Type":"ContainerDied","Data":"e9b7d4d66741bb06b457d99f674917560ba88991caff7b04d21b4ae1de93f96c"}
Sep 16 19:30:28.142432 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:28.142386    2561 scope.go:117] "RemoveContainer" containerID="4071390aee70580c3e2eeb6ce1c691db7fae43baebc6460941db0dce84bab18c"
Sep 16 19:30:28.153161 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:28.153136    2561 scope.go:117] "RemoveContainer" containerID="4071390aee70580c3e2eeb6ce1c691db7fae43baebc6460941db0dce84bab18c"
Sep 16 19:30:28.153448 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:30:28.153428    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4071390aee70580c3e2eeb6ce1c691db7fae43baebc6460941db0dce84bab18c\": container with ID starting with 4071390aee70580c3e2eeb6ce1c691db7fae43baebc6460941db0dce84bab18c not found: ID does not exist" containerID="4071390aee70580c3e2eeb6ce1c691db7fae43baebc6460941db0dce84bab18c"
Sep 16 19:30:28.153517 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:28.153463    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4071390aee70580c3e2eeb6ce1c691db7fae43baebc6460941db0dce84bab18c"} err="failed to get container status \"4071390aee70580c3e2eeb6ce1c691db7fae43baebc6460941db0dce84bab18c\": rpc error: code = NotFound desc = could not find container \"4071390aee70580c3e2eeb6ce1c691db7fae43baebc6460941db0dce84bab18c\": container with ID starting with 4071390aee70580c3e2eeb6ce1c691db7fae43baebc6460941db0dce84bab18c not found: ID does not exist"
Sep 16 19:30:28.167033 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:28.167002    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"]
Sep 16 19:30:28.172666 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:28.172638    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemx5j5"]
Sep 16 19:30:28.734162 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:28.734125    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="051c9cb7-e2ef-4518-9111-c860a9c8f1ae" path="/var/lib/kubelet/pods/051c9cb7-e2ef-4518-9111-c860a9c8f1ae/volumes"
Sep 16 19:30:32.360271 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.360242    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:32.459117 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.459050    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:32.545975 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.545932    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt" podUID="3e5704f0-871a-44aa-a49b-3d76203b385e" containerName="main" containerID="cri-o://90ee17f8bdccaa3e62255185dad79d136d128ddd8ded46398fe55132a45f4995" gracePeriod=45
Sep 16 19:30:32.804952 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.804926    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:30:32.886906 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.886870    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhrkl\" (UniqueName: \"kubernetes.io/projected/3e5704f0-871a-44aa-a49b-3d76203b385e-kube-api-access-dhrkl\") pod \"3e5704f0-871a-44aa-a49b-3d76203b385e\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") "
Sep 16 19:30:32.887116 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.886922    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-home\") pod \"3e5704f0-871a-44aa-a49b-3d76203b385e\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") "
Sep 16 19:30:32.887116 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.886950    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-model-cache\") pod \"3e5704f0-871a-44aa-a49b-3d76203b385e\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") "
Sep 16 19:30:32.887116 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.886966    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5704f0-871a-44aa-a49b-3d76203b385e-tls-certs\") pod \"3e5704f0-871a-44aa-a49b-3d76203b385e\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") "
Sep 16 19:30:32.887116 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.887022    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-dshm\") pod \"3e5704f0-871a-44aa-a49b-3d76203b385e\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") "
Sep 16 19:30:32.887116 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.887100    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-tmp-dir\") pod \"3e5704f0-871a-44aa-a49b-3d76203b385e\" (UID: \"3e5704f0-871a-44aa-a49b-3d76203b385e\") "
Sep 16 19:30:32.887405 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.887226    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-home" (OuterVolumeSpecName: "home") pod "3e5704f0-871a-44aa-a49b-3d76203b385e" (UID: "3e5704f0-871a-44aa-a49b-3d76203b385e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:32.887405 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.887259    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e5704f0-871a-44aa-a49b-3d76203b385e" (UID: "3e5704f0-871a-44aa-a49b-3d76203b385e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:32.887500 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.887412    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:30:32.887500 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.887426    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:30:32.887500 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.887449    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e5704f0-871a-44aa-a49b-3d76203b385e" (UID: "3e5704f0-871a-44aa-a49b-3d76203b385e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:32.889477 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.889449    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-dshm" (OuterVolumeSpecName: "dshm") pod "3e5704f0-871a-44aa-a49b-3d76203b385e" (UID: "3e5704f0-871a-44aa-a49b-3d76203b385e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:32.889587 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.889518    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e5704f0-871a-44aa-a49b-3d76203b385e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e5704f0-871a-44aa-a49b-3d76203b385e" (UID: "3e5704f0-871a-44aa-a49b-3d76203b385e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:32.889587 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.889541    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5704f0-871a-44aa-a49b-3d76203b385e-kube-api-access-dhrkl" (OuterVolumeSpecName: "kube-api-access-dhrkl") pod "3e5704f0-871a-44aa-a49b-3d76203b385e" (UID: "3e5704f0-871a-44aa-a49b-3d76203b385e"). InnerVolumeSpecName "kube-api-access-dhrkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:32.988478 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.988443    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:30:32.988478 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.988472    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e5704f0-871a-44aa-a49b-3d76203b385e-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:30:32.988478 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.988482    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhrkl\" (UniqueName: \"kubernetes.io/projected/3e5704f0-871a-44aa-a49b-3d76203b385e-kube-api-access-dhrkl\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:30:32.988714 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:32.988491    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5704f0-871a-44aa-a49b-3d76203b385e-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:30:33.173283 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:33.173246    2561 generic.go:356] "Generic (PLEG): container finished" podID="3e5704f0-871a-44aa-a49b-3d76203b385e" containerID="90ee17f8bdccaa3e62255185dad79d136d128ddd8ded46398fe55132a45f4995" exitCode=0
Sep 16 19:30:33.173471 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:33.173295    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"
Sep 16 19:30:33.173471 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:33.173304    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt" event={"ID":"3e5704f0-871a-44aa-a49b-3d76203b385e","Type":"ContainerDied","Data":"90ee17f8bdccaa3e62255185dad79d136d128ddd8ded46398fe55132a45f4995"}
Sep 16 19:30:33.173471 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:33.173357    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt" event={"ID":"3e5704f0-871a-44aa-a49b-3d76203b385e","Type":"ContainerDied","Data":"d8d505ffa2bd83ce289e95152ed3cd6f5fb2f70ceddd5fd5d455d0de9614d48c"}
Sep 16 19:30:33.173471 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:33.173391    2561 scope.go:117] "RemoveContainer" containerID="90ee17f8bdccaa3e62255185dad79d136d128ddd8ded46398fe55132a45f4995"
Sep 16 19:30:33.183306 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:33.183287    2561 scope.go:117] "RemoveContainer" containerID="90ee17f8bdccaa3e62255185dad79d136d128ddd8ded46398fe55132a45f4995"
Sep 16 19:30:33.183608 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:30:33.183588    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ee17f8bdccaa3e62255185dad79d136d128ddd8ded46398fe55132a45f4995\": container with ID starting with 90ee17f8bdccaa3e62255185dad79d136d128ddd8ded46398fe55132a45f4995 not found: ID does not exist" containerID="90ee17f8bdccaa3e62255185dad79d136d128ddd8ded46398fe55132a45f4995"
Sep 16 19:30:33.183656 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:33.183618    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ee17f8bdccaa3e62255185dad79d136d128ddd8ded46398fe55132a45f4995"} err="failed to get container status \"90ee17f8bdccaa3e62255185dad79d136d128ddd8ded46398fe55132a45f4995\": rpc error: code = NotFound desc = could not find container \"90ee17f8bdccaa3e62255185dad79d136d128ddd8ded46398fe55132a45f4995\": container with ID starting with 90ee17f8bdccaa3e62255185dad79d136d128ddd8ded46398fe55132a45f4995 not found: ID does not exist"
Sep 16 19:30:33.197430 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:33.197393    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"]
Sep 16 19:30:33.201778 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:33.201748    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zhdwt"]
Sep 16 19:30:34.735216 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:34.735183    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e5704f0-871a-44aa-a49b-3d76203b385e" path="/var/lib/kubelet/pods/3e5704f0-871a-44aa-a49b-3d76203b385e/volumes"
Sep 16 19:30:36.877468 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:36.877436    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:39.758136 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:39.758094    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:45.933120 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:45.933078    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:46.005869 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:46.005826    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:56.416679 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:56.416642    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:59.685051 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:30:59.685009    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:06.088088 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:06.088037    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:06.201316 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:06.201283    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:14.114832 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:14.114799    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:24.902016 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:24.901928    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:25.168575 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.168452    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"]
Sep 16 19:31:25.168915 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.168892    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e5704f0-871a-44aa-a49b-3d76203b385e" containerName="main"
Sep 16 19:31:25.168915 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.169000    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e5704f0-871a-44aa-a49b-3d76203b385e" containerName="main"
Sep 16 19:31:25.168915 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.169038    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="051c9cb7-e2ef-4518-9111-c860a9c8f1ae" containerName="main"
Sep 16 19:31:25.168915 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.169049    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="051c9cb7-e2ef-4518-9111-c860a9c8f1ae" containerName="main"
Sep 16 19:31:25.169559 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.169227    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="051c9cb7-e2ef-4518-9111-c860a9c8f1ae" containerName="main"
Sep 16 19:31:25.169559 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.169243    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e5704f0-871a-44aa-a49b-3d76203b385e" containerName="main"
Sep 16 19:31:25.172580 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.172560    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"
Sep 16 19:31:25.175599 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.175573    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 19:31:25.176988 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.176967    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-bcglm\""
Sep 16 19:31:25.176988 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.176978    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 19:31:25.177205 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.176978    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-5fjlw\""
Sep 16 19:31:25.177205 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.176978    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 19:31:25.183630 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.183603    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"]
Sep 16 19:31:25.199328 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.199279    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:25.272721 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.272669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x87hx\" (UniqueName: \"kubernetes.io/projected/e5806332-98b2-45a2-ab46-fd449f465a8e-kube-api-access-x87hx\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s\" (UID: \"e5806332-98b2-45a2-ab46-fd449f465a8e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"
Sep 16 19:31:25.272921 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.272786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5806332-98b2-45a2-ab46-fd449f465a8e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s\" (UID: \"e5806332-98b2-45a2-ab46-fd449f465a8e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"
Sep 16 19:31:25.374318 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.374273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5806332-98b2-45a2-ab46-fd449f465a8e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s\" (UID: \"e5806332-98b2-45a2-ab46-fd449f465a8e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"
Sep 16 19:31:25.374552 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.374346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x87hx\" (UniqueName: \"kubernetes.io/projected/e5806332-98b2-45a2-ab46-fd449f465a8e-kube-api-access-x87hx\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s\" (UID: \"e5806332-98b2-45a2-ab46-fd449f465a8e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"
Sep 16 19:31:25.377191 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.377165    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5806332-98b2-45a2-ab46-fd449f465a8e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s\" (UID: \"e5806332-98b2-45a2-ab46-fd449f465a8e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"
Sep 16 19:31:25.384248 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.384132    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x87hx\" (UniqueName: \"kubernetes.io/projected/e5806332-98b2-45a2-ab46-fd449f465a8e-kube-api-access-x87hx\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s\" (UID: \"e5806332-98b2-45a2-ab46-fd449f465a8e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"
Sep 16 19:31:25.485654 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.485620    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"
Sep 16 19:31:25.765104 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:25.765072    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"]
Sep 16 19:31:25.766251 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:31:25.766221    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5806332_98b2_45a2_ab46_fd449f465a8e.slice/crio-0b289dd007efd18e13539c3c44df1782f8dcafd9d3a3544abafbae351e8a34c9 WatchSource:0}: Error finding container 0b289dd007efd18e13539c3c44df1782f8dcafd9d3a3544abafbae351e8a34c9: Status 404 returned error can't find the container with id 0b289dd007efd18e13539c3c44df1782f8dcafd9d3a3544abafbae351e8a34c9
Sep 16 19:31:26.465826 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:26.465790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s" event={"ID":"e5806332-98b2-45a2-ab46-fd449f465a8e","Type":"ContainerStarted","Data":"92b2b3d8fa4d839e004289432ac60235bd6c6265ccc7cddeebc8307f0c095cd2"}
Sep 16 19:31:26.466269 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:26.465850    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s" event={"ID":"e5806332-98b2-45a2-ab46-fd449f465a8e","Type":"ContainerStarted","Data":"0b289dd007efd18e13539c3c44df1782f8dcafd9d3a3544abafbae351e8a34c9"}
Sep 16 19:31:26.466269 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:26.465875    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"
Sep 16 19:31:26.485316 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:26.485233    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s" podStartSLOduration=1.485210131 podStartE2EDuration="1.485210131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:26.483627622 +0000 UTC m=+2786.286345839" watchObservedRunningTime="2026-09-16 19:31:26.485210131 +0000 UTC m=+2786.287928348"
Sep 16 19:31:57.474386 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:31:57.474354    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"
Sep 16 19:32:06.151440 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:32:06.151395    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:28.356918 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:32:28.356878    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:50.037510 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:32:50.037472    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:50.137752 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:32:50.137709    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:50.462438 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:32:50.462378    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:50.623888 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:32:50.623797    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:54.235335 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:32:54.235297    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:55.201174 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:32:55.201140    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:19.897318 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:33:19.897287    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:33.268896 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:33:33.268861    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:39.336194 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:33:39.336155    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:39.436530 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:33:39.436465    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:46.245659 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:33:46.245622    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:47.994621 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:33:47.994591    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:48.004906 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:33:48.004876    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"]
Sep 16 19:33:48.072088 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:33:48.072022    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:57.474331 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:33:57.474295    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s" podUID="e5806332-98b2-45a2-ab46-fd449f465a8e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:03.024695 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:03.024647    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s" podUID="e5806332-98b2-45a2-ab46-fd449f465a8e" containerName="main" containerID="cri-o://92b2b3d8fa4d839e004289432ac60235bd6c6265ccc7cddeebc8307f0c095cd2" gracePeriod=45
Sep 16 19:34:07.475040 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:07.474995    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s" podUID="e5806332-98b2-45a2-ab46-fd449f465a8e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:17.474950 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:17.474906    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s" podUID="e5806332-98b2-45a2-ab46-fd449f465a8e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:17.475468 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:17.475020    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"
Sep 16 19:34:27.475291 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:27.475203    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s" podUID="e5806332-98b2-45a2-ab46-fd449f465a8e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:27.868473 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:27.868388    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:33.284040 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.284011    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"
Sep 16 19:34:33.440581 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.440481    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:33.464553 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.464520    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5806332-98b2-45a2-ab46-fd449f465a8e-tls-certs\") pod \"e5806332-98b2-45a2-ab46-fd449f465a8e\" (UID: \"e5806332-98b2-45a2-ab46-fd449f465a8e\") "
Sep 16 19:34:33.464665 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.464571    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x87hx\" (UniqueName: \"kubernetes.io/projected/e5806332-98b2-45a2-ab46-fd449f465a8e-kube-api-access-x87hx\") pod \"e5806332-98b2-45a2-ab46-fd449f465a8e\" (UID: \"e5806332-98b2-45a2-ab46-fd449f465a8e\") "
Sep 16 19:34:33.467090 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.467030    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5806332-98b2-45a2-ab46-fd449f465a8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5806332-98b2-45a2-ab46-fd449f465a8e" (UID: "e5806332-98b2-45a2-ab46-fd449f465a8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:33.467219 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.467173    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5806332-98b2-45a2-ab46-fd449f465a8e-kube-api-access-x87hx" (OuterVolumeSpecName: "kube-api-access-x87hx") pod "e5806332-98b2-45a2-ab46-fd449f465a8e" (UID: "e5806332-98b2-45a2-ab46-fd449f465a8e"). InnerVolumeSpecName "kube-api-access-x87hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:33.521668 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.521634    2561 generic.go:356] "Generic (PLEG): container finished" podID="e5806332-98b2-45a2-ab46-fd449f465a8e" containerID="92b2b3d8fa4d839e004289432ac60235bd6c6265ccc7cddeebc8307f0c095cd2" exitCode=0
Sep 16 19:34:33.521853 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.521679    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"
Sep 16 19:34:33.521853 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.521722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s" event={"ID":"e5806332-98b2-45a2-ab46-fd449f465a8e","Type":"ContainerDied","Data":"92b2b3d8fa4d839e004289432ac60235bd6c6265ccc7cddeebc8307f0c095cd2"}
Sep 16 19:34:33.521853 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.521778    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s" event={"ID":"e5806332-98b2-45a2-ab46-fd449f465a8e","Type":"ContainerDied","Data":"0b289dd007efd18e13539c3c44df1782f8dcafd9d3a3544abafbae351e8a34c9"}
Sep 16 19:34:33.521853 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.521803    2561 scope.go:117] "RemoveContainer" containerID="92b2b3d8fa4d839e004289432ac60235bd6c6265ccc7cddeebc8307f0c095cd2"
Sep 16 19:34:33.531924 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.531901    2561 scope.go:117] "RemoveContainer" containerID="92b2b3d8fa4d839e004289432ac60235bd6c6265ccc7cddeebc8307f0c095cd2"
Sep 16 19:34:33.532244 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:34:33.532224    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b2b3d8fa4d839e004289432ac60235bd6c6265ccc7cddeebc8307f0c095cd2\": container with ID starting with 92b2b3d8fa4d839e004289432ac60235bd6c6265ccc7cddeebc8307f0c095cd2 not found: ID does not exist" containerID="92b2b3d8fa4d839e004289432ac60235bd6c6265ccc7cddeebc8307f0c095cd2"
Sep 16 19:34:33.532303 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.532254    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b2b3d8fa4d839e004289432ac60235bd6c6265ccc7cddeebc8307f0c095cd2"} err="failed to get container status \"92b2b3d8fa4d839e004289432ac60235bd6c6265ccc7cddeebc8307f0c095cd2\": rpc error: code = NotFound desc = could not find container \"92b2b3d8fa4d839e004289432ac60235bd6c6265ccc7cddeebc8307f0c095cd2\": container with ID starting with 92b2b3d8fa4d839e004289432ac60235bd6c6265ccc7cddeebc8307f0c095cd2 not found: ID does not exist"
Sep 16 19:34:33.545115 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.545087    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"]
Sep 16 19:34:33.550501 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.550476    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejkb5s"]
Sep 16 19:34:33.565602 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.565575    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5806332-98b2-45a2-ab46-fd449f465a8e-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:34:33.565602 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:33.565604    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x87hx\" (UniqueName: \"kubernetes.io/projected/e5806332-98b2-45a2-ab46-fd449f465a8e-kube-api-access-x87hx\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:34:34.734278 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:34.734235    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5806332-98b2-45a2-ab46-fd449f465a8e" path="/var/lib/kubelet/pods/e5806332-98b2-45a2-ab46-fd449f465a8e/volumes"
Sep 16 19:34:34.865680 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:34.865647    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:34.975443 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:34.975409    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:40.390761 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.390720    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:40.583308 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.583274    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"]
Sep 16 19:34:40.583714 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.583698    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5806332-98b2-45a2-ab46-fd449f465a8e" containerName="main"
Sep 16 19:34:40.583714 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.583714    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5806332-98b2-45a2-ab46-fd449f465a8e" containerName="main"
Sep 16 19:34:40.583864 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.583793    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5806332-98b2-45a2-ab46-fd449f465a8e" containerName="main"
Sep 16 19:34:40.587000 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.586979    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"
Sep 16 19:34:40.596899 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.596877    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-vljlk\""
Sep 16 19:34:40.597977 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.597958    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 19:34:40.598135 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.598116    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 19:34:40.598218 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.598190    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 19:34:40.598218 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.598205    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-sxpkm\""
Sep 16 19:34:40.602569 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.602547    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"]
Sep 16 19:34:40.631664 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.631087    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62ksw\" (UniqueName: \"kubernetes.io/projected/9b002492-a7cb-439f-8369-6097cf4bec0c-kube-api-access-62ksw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf\" (UID: \"9b002492-a7cb-439f-8369-6097cf4bec0c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"
Sep 16 19:34:40.631664 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.631200    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b002492-a7cb-439f-8369-6097cf4bec0c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf\" (UID: \"9b002492-a7cb-439f-8369-6097cf4bec0c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"
Sep 16 19:34:40.635817 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.635790    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:40.731794 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.731752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62ksw\" (UniqueName: \"kubernetes.io/projected/9b002492-a7cb-439f-8369-6097cf4bec0c-kube-api-access-62ksw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf\" (UID: \"9b002492-a7cb-439f-8369-6097cf4bec0c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"
Sep 16 19:34:40.732012 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.731873    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b002492-a7cb-439f-8369-6097cf4bec0c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf\" (UID: \"9b002492-a7cb-439f-8369-6097cf4bec0c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"
Sep 16 19:34:40.734944 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.734921    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b002492-a7cb-439f-8369-6097cf4bec0c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf\" (UID: \"9b002492-a7cb-439f-8369-6097cf4bec0c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"
Sep 16 19:34:40.740372 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.740345    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62ksw\" (UniqueName: \"kubernetes.io/projected/9b002492-a7cb-439f-8369-6097cf4bec0c-kube-api-access-62ksw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf\" (UID: \"9b002492-a7cb-439f-8369-6097cf4bec0c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"
Sep 16 19:34:40.897221 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:40.897179    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"
Sep 16 19:34:41.192788 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:41.192745    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"]
Sep 16 19:34:41.574641 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:41.574600    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf" event={"ID":"9b002492-a7cb-439f-8369-6097cf4bec0c","Type":"ContainerStarted","Data":"935bcd826ef8e3c69dd9b2a3fa250d2ee3280071e3a2f9dd7185c404ae10d53c"}
Sep 16 19:34:41.575133 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:41.574661    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"
Sep 16 19:34:41.575133 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:41.574678    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf" event={"ID":"9b002492-a7cb-439f-8369-6097cf4bec0c","Type":"ContainerStarted","Data":"f8b5ab822134d6e5b1fde737ef74ef8ed5f7baf1ffd09b92caebd6b366658fdd"}
Sep 16 19:34:41.597487 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:41.597421    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf" podStartSLOduration=1.5974019720000001 podStartE2EDuration="1.597401972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:41.594180042 +0000 UTC m=+2981.396898259" watchObservedRunningTime="2026-09-16 19:34:41.597401972 +0000 UTC m=+2981.400120187"
Sep 16 19:34:42.406522 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:42.406476    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:48.441126 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:48.441085    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:48.517504 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:48.517471    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:55.008162 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:34:55.008119    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:01.022535 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:35:01.022508    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:35:01.033347 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:35:01.033324    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:35:12.582766 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:35:12.582730    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"
Sep 16 19:35:41.462842 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:35:41.462805    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:58.675335 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:35:58.675304    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:58.735364 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:35:58.735330    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:02.582083 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:02.581621    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:02.584180 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:02.584140    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf" podUID="9b002492-a7cb-439f-8369-6097cf4bec0c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:36:02.597673 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:02.597634    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"]
Sep 16 19:36:02.678773 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:02.678729    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:12.582975 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:12.582934    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf" podUID="9b002492-a7cb-439f-8369-6097cf4bec0c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:36:17.619655 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:17.619603    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf" podUID="9b002492-a7cb-439f-8369-6097cf4bec0c" containerName="main" containerID="cri-o://935bcd826ef8e3c69dd9b2a3fa250d2ee3280071e3a2f9dd7185c404ae10d53c" gracePeriod=45
Sep 16 19:36:22.589550 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:22.589503    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf" podUID="9b002492-a7cb-439f-8369-6097cf4bec0c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:36:22.590169 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:22.589639    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"
Sep 16 19:36:32.582367 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:32.582326    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf" podUID="9b002492-a7cb-439f-8369-6097cf4bec0c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:36:42.582531 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:42.582485    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf" podUID="9b002492-a7cb-439f-8369-6097cf4bec0c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:36:45.412167 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:45.412122    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:47.884049 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:47.883980    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"
Sep 16 19:36:47.983400 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:47.983359    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b002492-a7cb-439f-8369-6097cf4bec0c-tls-certs\") pod \"9b002492-a7cb-439f-8369-6097cf4bec0c\" (UID: \"9b002492-a7cb-439f-8369-6097cf4bec0c\") "
Sep 16 19:36:47.983583 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:47.983491    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62ksw\" (UniqueName: \"kubernetes.io/projected/9b002492-a7cb-439f-8369-6097cf4bec0c-kube-api-access-62ksw\") pod \"9b002492-a7cb-439f-8369-6097cf4bec0c\" (UID: \"9b002492-a7cb-439f-8369-6097cf4bec0c\") "
Sep 16 19:36:47.985816 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:47.985784    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b002492-a7cb-439f-8369-6097cf4bec0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b002492-a7cb-439f-8369-6097cf4bec0c" (UID: "9b002492-a7cb-439f-8369-6097cf4bec0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:36:47.985816 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:47.985800    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b002492-a7cb-439f-8369-6097cf4bec0c-kube-api-access-62ksw" (OuterVolumeSpecName: "kube-api-access-62ksw") pod "9b002492-a7cb-439f-8369-6097cf4bec0c" (UID: "9b002492-a7cb-439f-8369-6097cf4bec0c"). InnerVolumeSpecName "kube-api-access-62ksw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:36:48.084267 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:48.084233    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62ksw\" (UniqueName: \"kubernetes.io/projected/9b002492-a7cb-439f-8369-6097cf4bec0c-kube-api-access-62ksw\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:36:48.084267 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:48.084263    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b002492-a7cb-439f-8369-6097cf4bec0c-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:36:48.271854 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:48.271815    2561 generic.go:356] "Generic (PLEG): container finished" podID="9b002492-a7cb-439f-8369-6097cf4bec0c" containerID="935bcd826ef8e3c69dd9b2a3fa250d2ee3280071e3a2f9dd7185c404ae10d53c" exitCode=0
Sep 16 19:36:48.272032 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:48.271863    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"
Sep 16 19:36:48.272032 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:48.271883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf" event={"ID":"9b002492-a7cb-439f-8369-6097cf4bec0c","Type":"ContainerDied","Data":"935bcd826ef8e3c69dd9b2a3fa250d2ee3280071e3a2f9dd7185c404ae10d53c"}
Sep 16 19:36:48.272032 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:48.271937    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf" event={"ID":"9b002492-a7cb-439f-8369-6097cf4bec0c","Type":"ContainerDied","Data":"f8b5ab822134d6e5b1fde737ef74ef8ed5f7baf1ffd09b92caebd6b366658fdd"}
Sep 16 19:36:48.272032 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:48.271965    2561 scope.go:117] "RemoveContainer" containerID="935bcd826ef8e3c69dd9b2a3fa250d2ee3280071e3a2f9dd7185c404ae10d53c"
Sep 16 19:36:48.281966 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:48.281945    2561 scope.go:117] "RemoveContainer" containerID="935bcd826ef8e3c69dd9b2a3fa250d2ee3280071e3a2f9dd7185c404ae10d53c"
Sep 16 19:36:48.282294 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:36:48.282275    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935bcd826ef8e3c69dd9b2a3fa250d2ee3280071e3a2f9dd7185c404ae10d53c\": container with ID starting with 935bcd826ef8e3c69dd9b2a3fa250d2ee3280071e3a2f9dd7185c404ae10d53c not found: ID does not exist" containerID="935bcd826ef8e3c69dd9b2a3fa250d2ee3280071e3a2f9dd7185c404ae10d53c"
Sep 16 19:36:48.282343 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:48.282306    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935bcd826ef8e3c69dd9b2a3fa250d2ee3280071e3a2f9dd7185c404ae10d53c"} err="failed to get container status \"935bcd826ef8e3c69dd9b2a3fa250d2ee3280071e3a2f9dd7185c404ae10d53c\": rpc error: code = NotFound desc = could not find container \"935bcd826ef8e3c69dd9b2a3fa250d2ee3280071e3a2f9dd7185c404ae10d53c\": container with ID starting with 935bcd826ef8e3c69dd9b2a3fa250d2ee3280071e3a2f9dd7185c404ae10d53c not found: ID does not exist"
Sep 16 19:36:48.300447 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:48.300415    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"]
Sep 16 19:36:48.308006 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:48.307976    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche4rprf"]
Sep 16 19:36:48.736193 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:48.736158    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b002492-a7cb-439f-8369-6097cf4bec0c" path="/var/lib/kubelet/pods/9b002492-a7cb-439f-8369-6097cf4bec0c/volumes"
Sep 16 19:36:52.209176 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:52.209135    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:52.311316 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:52.311282    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:52.413926 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:52.413880    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:54.479900 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:54.479863    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:54.585481 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:54.585445    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:54.687316 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:54.687287    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:59.184019 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:36:59.183977    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:00.739045 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:00.738992    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:00.823007 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:00.822972    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:05.286926 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:05.286887    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:05.392678 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:05.392640    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:12.909149 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:12.909106    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:13.095973 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.095937    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"]
Sep 16 19:37:13.096526 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.096502    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b002492-a7cb-439f-8369-6097cf4bec0c" containerName="main"
Sep 16 19:37:13.096526 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.096528    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b002492-a7cb-439f-8369-6097cf4bec0c" containerName="main"
Sep 16 19:37:13.096715 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.096619    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b002492-a7cb-439f-8369-6097cf4bec0c" containerName="main"
Sep 16 19:37:13.100107 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.100089    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"
Sep 16 19:37:13.104078 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.104030    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-75rzh\""
Sep 16 19:37:13.104220 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.104103    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 16 19:37:13.104220 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.104044    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 16 19:37:13.104220 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.104135    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 16 19:37:13.104220 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.104112    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-gs42r\""
Sep 16 19:37:13.113932 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.113909    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"]
Sep 16 19:37:13.208745 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.208648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd71b18-0636-4e3c-ad0f-11d6f90e1d48-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r\" (UID: \"7fd71b18-0636-4e3c-ad0f-11d6f90e1d48\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"
Sep 16 19:37:13.208745 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.208712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjcjr\" (UniqueName: \"kubernetes.io/projected/7fd71b18-0636-4e3c-ad0f-11d6f90e1d48-kube-api-access-vjcjr\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r\" (UID: \"7fd71b18-0636-4e3c-ad0f-11d6f90e1d48\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"
Sep 16 19:37:13.309829 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.309785    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd71b18-0636-4e3c-ad0f-11d6f90e1d48-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r\" (UID: \"7fd71b18-0636-4e3c-ad0f-11d6f90e1d48\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"
Sep 16 19:37:13.310030 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.309860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjcjr\" (UniqueName: \"kubernetes.io/projected/7fd71b18-0636-4e3c-ad0f-11d6f90e1d48-kube-api-access-vjcjr\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r\" (UID: \"7fd71b18-0636-4e3c-ad0f-11d6f90e1d48\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"
Sep 16 19:37:13.312576 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.312547    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd71b18-0636-4e3c-ad0f-11d6f90e1d48-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r\" (UID: \"7fd71b18-0636-4e3c-ad0f-11d6f90e1d48\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"
Sep 16 19:37:13.320608 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.320580    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjcjr\" (UniqueName: \"kubernetes.io/projected/7fd71b18-0636-4e3c-ad0f-11d6f90e1d48-kube-api-access-vjcjr\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r\" (UID: \"7fd71b18-0636-4e3c-ad0f-11d6f90e1d48\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"
Sep 16 19:37:13.412041 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.412001    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"
Sep 16 19:37:13.704029 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:13.703992    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"]
Sep 16 19:37:13.706150 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:37:13.706115    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd71b18_0636_4e3c_ad0f_11d6f90e1d48.slice/crio-a3435471ed22546e66c9a81c77ddc1e5e0b5f2780ee453c866b8807f15822cbc WatchSource:0}: Error finding container a3435471ed22546e66c9a81c77ddc1e5e0b5f2780ee453c866b8807f15822cbc: Status 404 returned error can't find the container with id a3435471ed22546e66c9a81c77ddc1e5e0b5f2780ee453c866b8807f15822cbc
Sep 16 19:37:14.422237 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:14.422200    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r" event={"ID":"7fd71b18-0636-4e3c-ad0f-11d6f90e1d48","Type":"ContainerStarted","Data":"3f31237f7df904f960d2c3d0d12d6b427b16f4caa49afb7e1f461a3540d81d8e"}
Sep 16 19:37:14.422666 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:14.422252    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r" event={"ID":"7fd71b18-0636-4e3c-ad0f-11d6f90e1d48","Type":"ContainerStarted","Data":"a3435471ed22546e66c9a81c77ddc1e5e0b5f2780ee453c866b8807f15822cbc"}
Sep 16 19:37:14.422666 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:14.422382    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"
Sep 16 19:37:14.463769 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:14.463695    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r" podStartSLOduration=1.463676188 podStartE2EDuration="1.463676188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:14.462199774 +0000 UTC m=+3134.264917991" watchObservedRunningTime="2026-09-16 19:37:14.463676188 +0000 UTC m=+3134.266394454"
Sep 16 19:37:14.515334 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:14.515293    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:15.742772 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:15.742738    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"]
Sep 16 19:37:15.755450 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:15.755420    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:15.856261 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:15.856230    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:17.135228 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:17.135174    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:17.385779 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:17.385681    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:21.746100 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:21.745972    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:21.931480 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:21.931439    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:22.267434 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:22.267400    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:26.277281 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:26.277243    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:26.457787 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:26.457752    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:26.612682 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:26.612605    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:28.404536 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:28.404503    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:28.506466 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:28.506432    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:31.453897 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:31.453853    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r" podUID="7fd71b18-0636-4e3c-ad0f-11d6f90e1d48" containerName="main" containerID="cri-o://3f31237f7df904f960d2c3d0d12d6b427b16f4caa49afb7e1f461a3540d81d8e" gracePeriod=45
Sep 16 19:37:32.975111 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:32.975049    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:33.114872 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:33.114839    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:40.487765 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:40.487716    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:40.712031 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:40.711999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:42.692697 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:42.692655    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:42.765251 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:42.765216    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:45.079779 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:45.079741    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:45.226031 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:45.225999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:46.436760 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:46.436719    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r" podUID="7fd71b18-0636-4e3c-ad0f-11d6f90e1d48" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:50.134563 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:50.134525    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:51.149278 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.149246    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"]
Sep 16 19:37:51.153113 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.153087    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.156781 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.156754    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 19:37:51.156921 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.156844    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:37:51.156921 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.156846    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 19:37:51.156921 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.156894    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-8c7qs\""
Sep 16 19:37:51.156921 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.156895    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 19:37:51.157246 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.157228    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 19:37:51.169090 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.169040    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"]
Sep 16 19:37:51.178262 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.178233    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:51.258685 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.258651    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-home\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.258685 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.258688    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-kserve-provision-location\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.258939 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.258725    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l79jx\" (UniqueName: \"kubernetes.io/projected/0f7c6262-7b93-4580-973e-4c244550a1eb-kube-api-access-l79jx\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.258939 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.258747    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f7c6262-7b93-4580-973e-4c244550a1eb-cabundle-cert\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.258939 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.258764    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-tmp-dir\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.258939 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.258781    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-model-cache\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.258939 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.258806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f7c6262-7b93-4580-973e-4c244550a1eb-tls-certs\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.258939 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.258873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-dshm\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.360091 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.360016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-home\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.360091 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.360094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-kserve-provision-location\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.360359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.360156    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l79jx\" (UniqueName: \"kubernetes.io/projected/0f7c6262-7b93-4580-973e-4c244550a1eb-kube-api-access-l79jx\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.360359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.360183    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f7c6262-7b93-4580-973e-4c244550a1eb-cabundle-cert\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.360359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.360209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-tmp-dir\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.360359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.360235    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-model-cache\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.360359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.360276    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f7c6262-7b93-4580-973e-4c244550a1eb-tls-certs\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.360359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.360348    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-dshm\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.360779 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.360487    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-home\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.360779 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.360540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-kserve-provision-location\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.360779 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.360607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-tmp-dir\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.360779 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.360758    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-model-cache\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.360935 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.360845    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f7c6262-7b93-4580-973e-4c244550a1eb-cabundle-cert\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.362848 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.362828    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-dshm\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.362942 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.362918    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f7c6262-7b93-4580-973e-4c244550a1eb-tls-certs\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.372758 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.372727    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l79jx\" (UniqueName: \"kubernetes.io/projected/0f7c6262-7b93-4580-973e-4c244550a1eb-kube-api-access-l79jx\") pod \"stop-feature-test-kserve-6b464ddf66-nzplj\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.442050 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.441953    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:51.464172 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.464131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:51.754346 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:51.754312    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"]
Sep 16 19:37:51.756079 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:37:51.756034    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7c6262_7b93_4580_973e_4c244550a1eb.slice/crio-9756ea691477dea98170afacc574bae4cbfeb2e258b0c37f719e53eedfcfbf35 WatchSource:0}: Error finding container 9756ea691477dea98170afacc574bae4cbfeb2e258b0c37f719e53eedfcfbf35: Status 404 returned error can't find the container with id 9756ea691477dea98170afacc574bae4cbfeb2e258b0c37f719e53eedfcfbf35
Sep 16 19:37:52.641979 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:52.641936    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" event={"ID":"0f7c6262-7b93-4580-973e-4c244550a1eb","Type":"ContainerStarted","Data":"da324b0de5cf86919d65ac0a72873e9268dd8c00535b0837c417bb4aded7b568"}
Sep 16 19:37:52.642529 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:52.641993    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" event={"ID":"0f7c6262-7b93-4580-973e-4c244550a1eb","Type":"ContainerStarted","Data":"9756ea691477dea98170afacc574bae4cbfeb2e258b0c37f719e53eedfcfbf35"}
Sep 16 19:37:54.655408 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:54.655312    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerID="da324b0de5cf86919d65ac0a72873e9268dd8c00535b0837c417bb4aded7b568" exitCode=0
Sep 16 19:37:54.655408 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:54.655391    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" event={"ID":"0f7c6262-7b93-4580-973e-4c244550a1eb","Type":"ContainerDied","Data":"da324b0de5cf86919d65ac0a72873e9268dd8c00535b0837c417bb4aded7b568"}
Sep 16 19:37:55.663194 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:55.663157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" event={"ID":"0f7c6262-7b93-4580-973e-4c244550a1eb","Type":"ContainerStarted","Data":"72eb479f9254783441a70c657b224e0442819b7912df3de00f8d971b250e2dc9"}
Sep 16 19:37:55.705411 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:55.705355    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" podStartSLOduration=2.20223711 podStartE2EDuration="4.705338757s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.503101647s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:55.703323999 +0000 UTC m=+3175.506042238" watchObservedRunningTime="2026-09-16 19:37:55.705338757 +0000 UTC m=+3175.508056972"
Sep 16 19:37:56.435489 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:56.435447    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r" podUID="7fd71b18-0636-4e3c-ad0f-11d6f90e1d48" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:56.465168 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:56.465120    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:37:57.354361 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:37:57.354324    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:01.465210 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:01.465158    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:38:01.466664 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:01.466635    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 19:38:01.699492 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:01.699460    2561 generic.go:356] "Generic (PLEG): container finished" podID="7fd71b18-0636-4e3c-ad0f-11d6f90e1d48" containerID="3f31237f7df904f960d2c3d0d12d6b427b16f4caa49afb7e1f461a3540d81d8e" exitCode=0
Sep 16 19:38:01.699667 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:01.699534    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r" event={"ID":"7fd71b18-0636-4e3c-ad0f-11d6f90e1d48","Type":"ContainerDied","Data":"3f31237f7df904f960d2c3d0d12d6b427b16f4caa49afb7e1f461a3540d81d8e"}
Sep 16 19:38:01.725197 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:01.725172    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"
Sep 16 19:38:01.751832 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:01.751796    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjcjr\" (UniqueName: \"kubernetes.io/projected/7fd71b18-0636-4e3c-ad0f-11d6f90e1d48-kube-api-access-vjcjr\") pod \"7fd71b18-0636-4e3c-ad0f-11d6f90e1d48\" (UID: \"7fd71b18-0636-4e3c-ad0f-11d6f90e1d48\") "
Sep 16 19:38:01.752000 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:01.751863    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd71b18-0636-4e3c-ad0f-11d6f90e1d48-tls-certs\") pod \"7fd71b18-0636-4e3c-ad0f-11d6f90e1d48\" (UID: \"7fd71b18-0636-4e3c-ad0f-11d6f90e1d48\") "
Sep 16 19:38:01.754450 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:01.754412    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd71b18-0636-4e3c-ad0f-11d6f90e1d48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fd71b18-0636-4e3c-ad0f-11d6f90e1d48" (UID: "7fd71b18-0636-4e3c-ad0f-11d6f90e1d48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:01.754620 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:01.754443    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd71b18-0636-4e3c-ad0f-11d6f90e1d48-kube-api-access-vjcjr" (OuterVolumeSpecName: "kube-api-access-vjcjr") pod "7fd71b18-0636-4e3c-ad0f-11d6f90e1d48" (UID: "7fd71b18-0636-4e3c-ad0f-11d6f90e1d48"). InnerVolumeSpecName "kube-api-access-vjcjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:01.852879 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:01.852834    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjcjr\" (UniqueName: \"kubernetes.io/projected/7fd71b18-0636-4e3c-ad0f-11d6f90e1d48-kube-api-access-vjcjr\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:38:01.852879 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:01.852871    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd71b18-0636-4e3c-ad0f-11d6f90e1d48-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:38:02.709669 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:02.709633    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"
Sep 16 19:38:02.710143 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:02.709641    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r" event={"ID":"7fd71b18-0636-4e3c-ad0f-11d6f90e1d48","Type":"ContainerDied","Data":"a3435471ed22546e66c9a81c77ddc1e5e0b5f2780ee453c866b8807f15822cbc"}
Sep 16 19:38:02.710143 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:02.709802    2561 scope.go:117] "RemoveContainer" containerID="3f31237f7df904f960d2c3d0d12d6b427b16f4caa49afb7e1f461a3540d81d8e"
Sep 16 19:38:02.736789 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:02.736753    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"]
Sep 16 19:38:02.740228 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:02.740199    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-sm68r"]
Sep 16 19:38:04.735210 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:04.735172    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fd71b18-0636-4e3c-ad0f-11d6f90e1d48" path="/var/lib/kubelet/pods/7fd71b18-0636-4e3c-ad0f-11d6f90e1d48/volumes"
Sep 16 19:38:11.465497 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:11.465447    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 19:38:11.467101 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:11.467071    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:16.420282 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:16.420240    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:16.524349 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:16.524315    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:16.633559 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:16.633524    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:21.457533 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:21.457487    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:21.465621 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:21.465585    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 19:38:25.522804 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.522760    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:25.767489 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.767445    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"]
Sep 16 19:38:25.767935 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.767916    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd71b18-0636-4e3c-ad0f-11d6f90e1d48" containerName="main"
Sep 16 19:38:25.767935 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.767936    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd71b18-0636-4e3c-ad0f-11d6f90e1d48" containerName="main"
Sep 16 19:38:25.768118 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.768072    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd71b18-0636-4e3c-ad0f-11d6f90e1d48" containerName="main"
Sep 16 19:38:25.771561 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.771526    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"
Sep 16 19:38:25.775815 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.775749    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 19:38:25.775937 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.775869    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-f2c68\""
Sep 16 19:38:25.776007 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.775963    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 19:38:25.776007 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.775868    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 19:38:25.776166 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.775757    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-b95lc\""
Sep 16 19:38:25.787671 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.787639    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"]
Sep 16 19:38:25.808969 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.808935    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:25.893776 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.893729    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4f629a-0302-4730-ab75-4918d777aa5e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7\" (UID: \"2d4f629a-0302-4730-ab75-4918d777aa5e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"
Sep 16 19:38:25.893985 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.893947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p8bd\" (UniqueName: \"kubernetes.io/projected/2d4f629a-0302-4730-ab75-4918d777aa5e-kube-api-access-6p8bd\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7\" (UID: \"2d4f629a-0302-4730-ab75-4918d777aa5e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"
Sep 16 19:38:25.994799 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.994749    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p8bd\" (UniqueName: \"kubernetes.io/projected/2d4f629a-0302-4730-ab75-4918d777aa5e-kube-api-access-6p8bd\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7\" (UID: \"2d4f629a-0302-4730-ab75-4918d777aa5e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"
Sep 16 19:38:25.995008 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:25.994844    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4f629a-0302-4730-ab75-4918d777aa5e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7\" (UID: \"2d4f629a-0302-4730-ab75-4918d777aa5e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"
Sep 16 19:38:26.006808 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:26.006766    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4f629a-0302-4730-ab75-4918d777aa5e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7\" (UID: \"2d4f629a-0302-4730-ab75-4918d777aa5e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"
Sep 16 19:38:26.009462 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:26.009433    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p8bd\" (UniqueName: \"kubernetes.io/projected/2d4f629a-0302-4730-ab75-4918d777aa5e-kube-api-access-6p8bd\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7\" (UID: \"2d4f629a-0302-4730-ab75-4918d777aa5e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"
Sep 16 19:38:26.086797 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:26.086694    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"
Sep 16 19:38:26.374596 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:26.374565    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"]
Sep 16 19:38:26.376284 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:38:26.376253    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4f629a_0302_4730_ab75_4918d777aa5e.slice/crio-772451518b68b2d39e14a666fe8c3fe63bff41756c19e8da5c7653abb62a47a1 WatchSource:0}: Error finding container 772451518b68b2d39e14a666fe8c3fe63bff41756c19e8da5c7653abb62a47a1: Status 404 returned error can't find the container with id 772451518b68b2d39e14a666fe8c3fe63bff41756c19e8da5c7653abb62a47a1
Sep 16 19:38:26.855826 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:26.855776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7" event={"ID":"2d4f629a-0302-4730-ab75-4918d777aa5e","Type":"ContainerStarted","Data":"a800e99bf169d4f04e1561f8495419250d2167f7e47ae3a2cfcff2f2f1dcfe7d"}
Sep 16 19:38:26.856363 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:26.855840    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7" event={"ID":"2d4f629a-0302-4730-ab75-4918d777aa5e","Type":"ContainerStarted","Data":"772451518b68b2d39e14a666fe8c3fe63bff41756c19e8da5c7653abb62a47a1"}
Sep 16 19:38:26.856363 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:26.855965    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"
Sep 16 19:38:26.885177 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:26.885114    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7" podStartSLOduration=1.885092755 podStartE2EDuration="1.885092755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:38:26.884667884 +0000 UTC m=+3206.687386101" watchObservedRunningTime="2026-09-16 19:38:26.885092755 +0000 UTC m=+3206.687810983"
Sep 16 19:38:31.204245 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:31.204204    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:31.465488 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:31.465438    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 19:38:37.606422 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:37.606378    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:41.465181 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:41.465127    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 19:38:42.604549 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:42.604516    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:44.037656 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:44.037620    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:50.131503 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:50.131467    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:51.464909 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:51.464802    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 19:38:57.864650 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:57.864616    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"
Sep 16 19:38:58.581327 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:58.581289    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:58.881194 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:38:58.881164    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:01.465249 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:39:01.465200    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 19:39:05.830421 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:39:05.830385    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:05.905780 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:39:05.905741    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:11.465609 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:39:11.465550    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 19:39:21.465422 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:39:21.465366    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 19:39:31.465478 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:39:31.465424    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 19:39:41.465011 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:39:41.464951    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 19:39:51.475052 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:39:51.475019    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:39:51.483312 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:39:51.483282    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:39:53.178831 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:39:53.178795    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"]
Sep 16 19:39:53.207776 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:39:53.207743    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:53.340535 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:39:53.340494    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:53.531523 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:39:53.531486    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:01.063364 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:01.063331    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:40:01.076459 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:01.076439    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:40:08.415558 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:08.415510    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main" containerID="cri-o://72eb479f9254783441a70c657b224e0442819b7912df3de00f8d971b250e2dc9" gracePeriod=45
Sep 16 19:40:08.476376 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:08.476334    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 19:40:09.080770 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.080739    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:40:09.151969 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.151900    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-dshm\") pod \"0f7c6262-7b93-4580-973e-4c244550a1eb\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") "
Sep 16 19:40:09.151969 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.151955    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-model-cache\") pod \"0f7c6262-7b93-4580-973e-4c244550a1eb\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") "
Sep 16 19:40:09.152182 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.151996    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-kserve-provision-location\") pod \"0f7c6262-7b93-4580-973e-4c244550a1eb\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") "
Sep 16 19:40:09.152182 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.152036    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l79jx\" (UniqueName: \"kubernetes.io/projected/0f7c6262-7b93-4580-973e-4c244550a1eb-kube-api-access-l79jx\") pod \"0f7c6262-7b93-4580-973e-4c244550a1eb\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") "
Sep 16 19:40:09.152182 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.152086    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-home\") pod \"0f7c6262-7b93-4580-973e-4c244550a1eb\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") "
Sep 16 19:40:09.152182 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.152138    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f7c6262-7b93-4580-973e-4c244550a1eb-cabundle-cert\") pod \"0f7c6262-7b93-4580-973e-4c244550a1eb\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") "
Sep 16 19:40:09.152392 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.152255    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f7c6262-7b93-4580-973e-4c244550a1eb-tls-certs\") pod \"0f7c6262-7b93-4580-973e-4c244550a1eb\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") "
Sep 16 19:40:09.152392 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.152287    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f7c6262-7b93-4580-973e-4c244550a1eb" (UID: "0f7c6262-7b93-4580-973e-4c244550a1eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:09.152392 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.152304    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-tmp-dir\") pod \"0f7c6262-7b93-4580-973e-4c244550a1eb\" (UID: \"0f7c6262-7b93-4580-973e-4c244550a1eb\") "
Sep 16 19:40:09.152562 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.152516    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f7c6262-7b93-4580-973e-4c244550a1eb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0f7c6262-7b93-4580-973e-4c244550a1eb" (UID: "0f7c6262-7b93-4580-973e-4c244550a1eb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:40:09.152621 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.152570    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:40:09.152935 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.152898    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-home" (OuterVolumeSpecName: "home") pod "0f7c6262-7b93-4580-973e-4c244550a1eb" (UID: "0f7c6262-7b93-4580-973e-4c244550a1eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:09.154484 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.154455    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-dshm" (OuterVolumeSpecName: "dshm") pod "0f7c6262-7b93-4580-973e-4c244550a1eb" (UID: "0f7c6262-7b93-4580-973e-4c244550a1eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:09.154693 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.154672    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7c6262-7b93-4580-973e-4c244550a1eb-kube-api-access-l79jx" (OuterVolumeSpecName: "kube-api-access-l79jx") pod "0f7c6262-7b93-4580-973e-4c244550a1eb" (UID: "0f7c6262-7b93-4580-973e-4c244550a1eb"). InnerVolumeSpecName "kube-api-access-l79jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:40:09.154844 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.154829    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f7c6262-7b93-4580-973e-4c244550a1eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f7c6262-7b93-4580-973e-4c244550a1eb" (UID: "0f7c6262-7b93-4580-973e-4c244550a1eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:40:09.164910 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.164880    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f7c6262-7b93-4580-973e-4c244550a1eb" (UID: "0f7c6262-7b93-4580-973e-4c244550a1eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:09.173953 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.173930    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f7c6262-7b93-4580-973e-4c244550a1eb" (UID: "0f7c6262-7b93-4580-973e-4c244550a1eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:09.253652 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.253630    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-kserve-provision-location\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:40:09.253652 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.253649    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l79jx\" (UniqueName: \"kubernetes.io/projected/0f7c6262-7b93-4580-973e-4c244550a1eb-kube-api-access-l79jx\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:40:09.253783 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.253659    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:40:09.253783 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.253667    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f7c6262-7b93-4580-973e-4c244550a1eb-cabundle-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:40:09.253783 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.253676    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f7c6262-7b93-4580-973e-4c244550a1eb-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:40:09.253783 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.253683    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:40:09.253783 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.253690    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f7c6262-7b93-4580-973e-4c244550a1eb-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:40:09.480282 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.480243    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerID="72eb479f9254783441a70c657b224e0442819b7912df3de00f8d971b250e2dc9" exitCode=0
Sep 16 19:40:09.480746 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.480295    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"
Sep 16 19:40:09.480746 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.480333    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" event={"ID":"0f7c6262-7b93-4580-973e-4c244550a1eb","Type":"ContainerDied","Data":"72eb479f9254783441a70c657b224e0442819b7912df3de00f8d971b250e2dc9"}
Sep 16 19:40:09.480746 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.480389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj" event={"ID":"0f7c6262-7b93-4580-973e-4c244550a1eb","Type":"ContainerDied","Data":"9756ea691477dea98170afacc574bae4cbfeb2e258b0c37f719e53eedfcfbf35"}
Sep 16 19:40:09.480746 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.480414    2561 scope.go:117] "RemoveContainer" containerID="72eb479f9254783441a70c657b224e0442819b7912df3de00f8d971b250e2dc9"
Sep 16 19:40:09.490862 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.490844    2561 scope.go:117] "RemoveContainer" containerID="da324b0de5cf86919d65ac0a72873e9268dd8c00535b0837c417bb4aded7b568"
Sep 16 19:40:09.499001 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.498984    2561 scope.go:117] "RemoveContainer" containerID="72eb479f9254783441a70c657b224e0442819b7912df3de00f8d971b250e2dc9"
Sep 16 19:40:09.499281 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:40:09.499262    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72eb479f9254783441a70c657b224e0442819b7912df3de00f8d971b250e2dc9\": container with ID starting with 72eb479f9254783441a70c657b224e0442819b7912df3de00f8d971b250e2dc9 not found: ID does not exist" containerID="72eb479f9254783441a70c657b224e0442819b7912df3de00f8d971b250e2dc9"
Sep 16 19:40:09.499349 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.499291    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72eb479f9254783441a70c657b224e0442819b7912df3de00f8d971b250e2dc9"} err="failed to get container status \"72eb479f9254783441a70c657b224e0442819b7912df3de00f8d971b250e2dc9\": rpc error: code = NotFound desc = could not find container \"72eb479f9254783441a70c657b224e0442819b7912df3de00f8d971b250e2dc9\": container with ID starting with 72eb479f9254783441a70c657b224e0442819b7912df3de00f8d971b250e2dc9 not found: ID does not exist"
Sep 16 19:40:09.499349 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.499308    2561 scope.go:117] "RemoveContainer" containerID="da324b0de5cf86919d65ac0a72873e9268dd8c00535b0837c417bb4aded7b568"
Sep 16 19:40:09.499535 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:40:09.499521    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da324b0de5cf86919d65ac0a72873e9268dd8c00535b0837c417bb4aded7b568\": container with ID starting with da324b0de5cf86919d65ac0a72873e9268dd8c00535b0837c417bb4aded7b568 not found: ID does not exist" containerID="da324b0de5cf86919d65ac0a72873e9268dd8c00535b0837c417bb4aded7b568"
Sep 16 19:40:09.499573 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.499539    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da324b0de5cf86919d65ac0a72873e9268dd8c00535b0837c417bb4aded7b568"} err="failed to get container status \"da324b0de5cf86919d65ac0a72873e9268dd8c00535b0837c417bb4aded7b568\": rpc error: code = NotFound desc = could not find container \"da324b0de5cf86919d65ac0a72873e9268dd8c00535b0837c417bb4aded7b568\": container with ID starting with da324b0de5cf86919d65ac0a72873e9268dd8c00535b0837c417bb4aded7b568 not found: ID does not exist"
Sep 16 19:40:09.504486 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.504464    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"]
Sep 16 19:40:09.508531 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:09.508509    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-nzplj"]
Sep 16 19:40:10.734347 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:10.734313    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" path="/var/lib/kubelet/pods/0f7c6262-7b93-4580-973e-4c244550a1eb/volumes"
Sep 16 19:40:20.176498 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.176464    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"]
Sep 16 19:40:20.177111 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.177090    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="storage-initializer"
Sep 16 19:40:20.177111 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.177112    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="storage-initializer"
Sep 16 19:40:20.177264 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.177131    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main"
Sep 16 19:40:20.177264 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.177138    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main"
Sep 16 19:40:20.177264 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.177231    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f7c6262-7b93-4580-973e-4c244550a1eb" containerName="main"
Sep 16 19:40:20.180431 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.180410    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.183758 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.183736    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 19:40:20.183933 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.183917    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 19:40:20.184014 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.183971    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 19:40:20.184183 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.184164    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 19:40:20.184252 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.184214    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:40:20.184308 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.184284    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-8c7qs\""
Sep 16 19:40:20.194191 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.194170    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"]
Sep 16 19:40:20.213257 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.213235    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:20.250092 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.250048    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-model-cache\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.250215 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.250098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-kserve-provision-location\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.250215 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.250153    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-tls-certs\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.250315 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.250232    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-cabundle-cert\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.250315 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.250252    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-home\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.250315 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.250287    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-tmp-dir\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.250439 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.250319    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98jrd\" (UniqueName: \"kubernetes.io/projected/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-kube-api-access-98jrd\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.250439 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.250340    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-dshm\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.350802 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.350770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98jrd\" (UniqueName: \"kubernetes.io/projected/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-kube-api-access-98jrd\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.350975 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.350813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-dshm\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.350975 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.350850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-model-cache\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.350975 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.350872    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-kserve-provision-location\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.350975 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.350906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-tls-certs\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.351229 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.350986    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-cabundle-cert\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.351229 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.351015    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-home\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.351229 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.351115    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-tmp-dir\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.351375 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.351336    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-model-cache\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.351439 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.351362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-kserve-provision-location\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.351549 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.351526    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-home\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.351713 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.351527    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-tmp-dir\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.351971 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.351945    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-cabundle-cert\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.353431 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.353410    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-dshm\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.353682 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.353664    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-tls-certs\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.363402 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.363381    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98jrd\" (UniqueName: \"kubernetes.io/projected/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-kube-api-access-98jrd\") pod \"stop-feature-test-kserve-6b464ddf66-d55qq\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.365913 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.365896    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:20.491362 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.491335    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:20.774508 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:20.774478    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"]
Sep 16 19:40:20.776508 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:40:20.776476    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96fabfe_8b34_4c5b_89c5_3b33c15c295d.slice/crio-2ce20014143fd0febcfaac94ddb96ab30c99b42994af661b308bee880c859afc WatchSource:0}: Error finding container 2ce20014143fd0febcfaac94ddb96ab30c99b42994af661b308bee880c859afc: Status 404 returned error can't find the container with id 2ce20014143fd0febcfaac94ddb96ab30c99b42994af661b308bee880c859afc
Sep 16 19:40:21.558895 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:21.558858    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" event={"ID":"e96fabfe-8b34-4c5b-89c5-3b33c15c295d","Type":"ContainerStarted","Data":"4b619950214bdd6f437156aa8931b1140001beccc0b50b765ede6ad0f904accb"}
Sep 16 19:40:21.559425 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:21.558913    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" event={"ID":"e96fabfe-8b34-4c5b-89c5-3b33c15c295d","Type":"ContainerStarted","Data":"2ce20014143fd0febcfaac94ddb96ab30c99b42994af661b308bee880c859afc"}
Sep 16 19:40:23.571775 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:23.571739    2561 generic.go:356] "Generic (PLEG): container finished" podID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerID="4b619950214bdd6f437156aa8931b1140001beccc0b50b765ede6ad0f904accb" exitCode=0
Sep 16 19:40:23.572288 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:23.571819    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" event={"ID":"e96fabfe-8b34-4c5b-89c5-3b33c15c295d","Type":"ContainerDied","Data":"4b619950214bdd6f437156aa8931b1140001beccc0b50b765ede6ad0f904accb"}
Sep 16 19:40:24.579629 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:24.579593    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" event={"ID":"e96fabfe-8b34-4c5b-89c5-3b33c15c295d","Type":"ContainerStarted","Data":"fe413fe51c00162e1a11f349a212aa62bf2d4e213511a2115463916c47998fcb"}
Sep 16 19:40:24.607235 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:24.607181    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" podStartSLOduration=2.333856667 podStartE2EDuration="4.607164537s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.27330787s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:24.604323646 +0000 UTC m=+3324.407041862" watchObservedRunningTime="2026-09-16 19:40:24.607164537 +0000 UTC m=+3324.409882753"
Sep 16 19:40:25.492159 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:25.492116    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:27.805515 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:27.805474    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:30.491616 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:30.491582    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:40:30.493568 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:30.493537    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 19:40:33.202736 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:33.202695    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"]
Sep 16 19:40:33.204451 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:33.204405    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:33.269619 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:33.269589    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:37.865282 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:37.865240    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7" podUID="2d4f629a-0302-4730-ab75-4918d777aa5e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:40:40.492623 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:40.492575    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 19:40:47.864726 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:47.864685    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7" podUID="2d4f629a-0302-4730-ab75-4918d777aa5e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:40:48.224010 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:48.223937    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7" podUID="2d4f629a-0302-4730-ab75-4918d777aa5e" containerName="main" containerID="cri-o://a800e99bf169d4f04e1561f8495419250d2167f7e47ae3a2cfcff2f2f1dcfe7d" gracePeriod=45
Sep 16 19:40:50.492669 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:50.492618    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 19:40:57.865529 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:57.865485    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7" podUID="2d4f629a-0302-4730-ab75-4918d777aa5e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:40:57.866131 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:40:57.865601    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"
Sep 16 19:41:00.491917 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:00.491871    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 19:41:07.865729 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:07.865681    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7" podUID="2d4f629a-0302-4730-ab75-4918d777aa5e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:41:10.491796 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:10.491737    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 19:41:17.865653 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:17.865607    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7" podUID="2d4f629a-0302-4730-ab75-4918d777aa5e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:41:18.473985 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.473956    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"
Sep 16 19:41:18.605502 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.605417    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p8bd\" (UniqueName: \"kubernetes.io/projected/2d4f629a-0302-4730-ab75-4918d777aa5e-kube-api-access-6p8bd\") pod \"2d4f629a-0302-4730-ab75-4918d777aa5e\" (UID: \"2d4f629a-0302-4730-ab75-4918d777aa5e\") "
Sep 16 19:41:18.605502 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.605477    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4f629a-0302-4730-ab75-4918d777aa5e-tls-certs\") pod \"2d4f629a-0302-4730-ab75-4918d777aa5e\" (UID: \"2d4f629a-0302-4730-ab75-4918d777aa5e\") "
Sep 16 19:41:18.607837 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.607804    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d4f629a-0302-4730-ab75-4918d777aa5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d4f629a-0302-4730-ab75-4918d777aa5e" (UID: "2d4f629a-0302-4730-ab75-4918d777aa5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:41:18.607952 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.607837    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4f629a-0302-4730-ab75-4918d777aa5e-kube-api-access-6p8bd" (OuterVolumeSpecName: "kube-api-access-6p8bd") pod "2d4f629a-0302-4730-ab75-4918d777aa5e" (UID: "2d4f629a-0302-4730-ab75-4918d777aa5e"). InnerVolumeSpecName "kube-api-access-6p8bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:41:18.706817 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.706782    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p8bd\" (UniqueName: \"kubernetes.io/projected/2d4f629a-0302-4730-ab75-4918d777aa5e-kube-api-access-6p8bd\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:41:18.706817 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.706814    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4f629a-0302-4730-ab75-4918d777aa5e-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:41:18.892306 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.892212    2561 generic.go:356] "Generic (PLEG): container finished" podID="2d4f629a-0302-4730-ab75-4918d777aa5e" containerID="a800e99bf169d4f04e1561f8495419250d2167f7e47ae3a2cfcff2f2f1dcfe7d" exitCode=0
Sep 16 19:41:18.892306 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.892259    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"
Sep 16 19:41:18.892827 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.892248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7" event={"ID":"2d4f629a-0302-4730-ab75-4918d777aa5e","Type":"ContainerDied","Data":"a800e99bf169d4f04e1561f8495419250d2167f7e47ae3a2cfcff2f2f1dcfe7d"}
Sep 16 19:41:18.892827 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.892394    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7" event={"ID":"2d4f629a-0302-4730-ab75-4918d777aa5e","Type":"ContainerDied","Data":"772451518b68b2d39e14a666fe8c3fe63bff41756c19e8da5c7653abb62a47a1"}
Sep 16 19:41:18.892827 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.892422    2561 scope.go:117] "RemoveContainer" containerID="a800e99bf169d4f04e1561f8495419250d2167f7e47ae3a2cfcff2f2f1dcfe7d"
Sep 16 19:41:18.903636 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.903617    2561 scope.go:117] "RemoveContainer" containerID="a800e99bf169d4f04e1561f8495419250d2167f7e47ae3a2cfcff2f2f1dcfe7d"
Sep 16 19:41:18.903958 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:41:18.903932    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a800e99bf169d4f04e1561f8495419250d2167f7e47ae3a2cfcff2f2f1dcfe7d\": container with ID starting with a800e99bf169d4f04e1561f8495419250d2167f7e47ae3a2cfcff2f2f1dcfe7d not found: ID does not exist" containerID="a800e99bf169d4f04e1561f8495419250d2167f7e47ae3a2cfcff2f2f1dcfe7d"
Sep 16 19:41:18.904008 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.903967    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a800e99bf169d4f04e1561f8495419250d2167f7e47ae3a2cfcff2f2f1dcfe7d"} err="failed to get container status \"a800e99bf169d4f04e1561f8495419250d2167f7e47ae3a2cfcff2f2f1dcfe7d\": rpc error: code = NotFound desc = could not find container \"a800e99bf169d4f04e1561f8495419250d2167f7e47ae3a2cfcff2f2f1dcfe7d\": container with ID starting with a800e99bf169d4f04e1561f8495419250d2167f7e47ae3a2cfcff2f2f1dcfe7d not found: ID does not exist"
Sep 16 19:41:18.915709 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.915648    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"]
Sep 16 19:41:18.919897 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:18.919873    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherbrb7"]
Sep 16 19:41:20.492490 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:20.492450    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 19:41:20.736324 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:20.736289    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d4f629a-0302-4730-ab75-4918d777aa5e" path="/var/lib/kubelet/pods/2d4f629a-0302-4730-ab75-4918d777aa5e/volumes"
Sep 16 19:41:23.276518 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:23.276483    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:23.371589 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:23.371552    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:29.508860 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:29.508802    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:30.491975 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:30.491932    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 19:41:34.941522 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:34.941485    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:35.155480 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:35.155447    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:35.960992 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:35.960956    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:36.046569 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:36.046528    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:40.491980 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:40.491927    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 19:41:47.339610 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:47.339578    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:50.492108 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:41:50.492051    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 19:42:00.491864 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:00.491823    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 19:42:10.492397 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:10.492355    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 19:42:20.501493 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:20.501461    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:42:20.509261 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:20.509240    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:42:22.284596 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:22.284551    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:22.301212 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:22.301184    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"]
Sep 16 19:42:23.877984 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:23.877950    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:37.324197 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:37.324131    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main" containerID="cri-o://fe413fe51c00162e1a11f349a212aa62bf2d4e213511a2115463916c47998fcb" gracePeriod=45
Sep 16 19:42:37.502843 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:37.502797    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 19:42:37.986393 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:37.986364    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:42:38.126405 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.126328    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-tls-certs\") pod \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") "
Sep 16 19:42:38.126405 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.126395    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98jrd\" (UniqueName: \"kubernetes.io/projected/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-kube-api-access-98jrd\") pod \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") "
Sep 16 19:42:38.126640 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.126442    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-kserve-provision-location\") pod \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") "
Sep 16 19:42:38.126640 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.126457    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-cabundle-cert\") pod \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") "
Sep 16 19:42:38.126640 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.126482    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-tmp-dir\") pod \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") "
Sep 16 19:42:38.126640 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.126525    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-home\") pod \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") "
Sep 16 19:42:38.126640 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.126547    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-model-cache\") pod \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") "
Sep 16 19:42:38.126640 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.126581    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-dshm\") pod \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\" (UID: \"e96fabfe-8b34-4c5b-89c5-3b33c15c295d\") "
Sep 16 19:42:38.127011 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.126984    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e96fabfe-8b34-4c5b-89c5-3b33c15c295d" (UID: "e96fabfe-8b34-4c5b-89c5-3b33c15c295d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:42:38.127294 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.127102    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e96fabfe-8b34-4c5b-89c5-3b33c15c295d" (UID: "e96fabfe-8b34-4c5b-89c5-3b33c15c295d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.127473 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.127440    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-home" (OuterVolumeSpecName: "home") pod "e96fabfe-8b34-4c5b-89c5-3b33c15c295d" (UID: "e96fabfe-8b34-4c5b-89c5-3b33c15c295d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.128808 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.128787    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e96fabfe-8b34-4c5b-89c5-3b33c15c295d" (UID: "e96fabfe-8b34-4c5b-89c5-3b33c15c295d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:42:38.128808 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.128810    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-kube-api-access-98jrd" (OuterVolumeSpecName: "kube-api-access-98jrd") pod "e96fabfe-8b34-4c5b-89c5-3b33c15c295d" (UID: "e96fabfe-8b34-4c5b-89c5-3b33c15c295d"). InnerVolumeSpecName "kube-api-access-98jrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:38.129262 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.129246    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-dshm" (OuterVolumeSpecName: "dshm") pod "e96fabfe-8b34-4c5b-89c5-3b33c15c295d" (UID: "e96fabfe-8b34-4c5b-89c5-3b33c15c295d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.138634 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.138610    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e96fabfe-8b34-4c5b-89c5-3b33c15c295d" (UID: "e96fabfe-8b34-4c5b-89c5-3b33c15c295d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.155526 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.155503    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e96fabfe-8b34-4c5b-89c5-3b33c15c295d" (UID: "e96fabfe-8b34-4c5b-89c5-3b33c15c295d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.228236 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.228211    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98jrd\" (UniqueName: \"kubernetes.io/projected/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-kube-api-access-98jrd\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.228236 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.228233    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-kserve-provision-location\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.228368 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.228243    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-cabundle-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.228368 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.228252    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.228368 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.228259    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.228368 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.228266    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.228368 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.228275    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.228368 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.228281    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96fabfe-8b34-4c5b-89c5-3b33c15c295d-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.373034 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.373000    2561 generic.go:356] "Generic (PLEG): container finished" podID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerID="fe413fe51c00162e1a11f349a212aa62bf2d4e213511a2115463916c47998fcb" exitCode=0
Sep 16 19:42:38.373485 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.373077    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"
Sep 16 19:42:38.373485 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.373087    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" event={"ID":"e96fabfe-8b34-4c5b-89c5-3b33c15c295d","Type":"ContainerDied","Data":"fe413fe51c00162e1a11f349a212aa62bf2d4e213511a2115463916c47998fcb"}
Sep 16 19:42:38.373485 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.373148    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq" event={"ID":"e96fabfe-8b34-4c5b-89c5-3b33c15c295d","Type":"ContainerDied","Data":"2ce20014143fd0febcfaac94ddb96ab30c99b42994af661b308bee880c859afc"}
Sep 16 19:42:38.373485 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.373174    2561 scope.go:117] "RemoveContainer" containerID="fe413fe51c00162e1a11f349a212aa62bf2d4e213511a2115463916c47998fcb"
Sep 16 19:42:38.383225 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.383206    2561 scope.go:117] "RemoveContainer" containerID="4b619950214bdd6f437156aa8931b1140001beccc0b50b765ede6ad0f904accb"
Sep 16 19:42:38.391310 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.391294    2561 scope.go:117] "RemoveContainer" containerID="fe413fe51c00162e1a11f349a212aa62bf2d4e213511a2115463916c47998fcb"
Sep 16 19:42:38.391565 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:42:38.391548    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe413fe51c00162e1a11f349a212aa62bf2d4e213511a2115463916c47998fcb\": container with ID starting with fe413fe51c00162e1a11f349a212aa62bf2d4e213511a2115463916c47998fcb not found: ID does not exist" containerID="fe413fe51c00162e1a11f349a212aa62bf2d4e213511a2115463916c47998fcb"
Sep 16 19:42:38.391616 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.391575    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe413fe51c00162e1a11f349a212aa62bf2d4e213511a2115463916c47998fcb"} err="failed to get container status \"fe413fe51c00162e1a11f349a212aa62bf2d4e213511a2115463916c47998fcb\": rpc error: code = NotFound desc = could not find container \"fe413fe51c00162e1a11f349a212aa62bf2d4e213511a2115463916c47998fcb\": container with ID starting with fe413fe51c00162e1a11f349a212aa62bf2d4e213511a2115463916c47998fcb not found: ID does not exist"
Sep 16 19:42:38.391616 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.391593    2561 scope.go:117] "RemoveContainer" containerID="4b619950214bdd6f437156aa8931b1140001beccc0b50b765ede6ad0f904accb"
Sep 16 19:42:38.391857 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:42:38.391837    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b619950214bdd6f437156aa8931b1140001beccc0b50b765ede6ad0f904accb\": container with ID starting with 4b619950214bdd6f437156aa8931b1140001beccc0b50b765ede6ad0f904accb not found: ID does not exist" containerID="4b619950214bdd6f437156aa8931b1140001beccc0b50b765ede6ad0f904accb"
Sep 16 19:42:38.391905 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.391862    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b619950214bdd6f437156aa8931b1140001beccc0b50b765ede6ad0f904accb"} err="failed to get container status \"4b619950214bdd6f437156aa8931b1140001beccc0b50b765ede6ad0f904accb\": rpc error: code = NotFound desc = could not find container \"4b619950214bdd6f437156aa8931b1140001beccc0b50b765ede6ad0f904accb\": container with ID starting with 4b619950214bdd6f437156aa8931b1140001beccc0b50b765ede6ad0f904accb not found: ID does not exist"
Sep 16 19:42:38.400960 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.400935    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"]
Sep 16 19:42:38.405046 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.405025    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6b464ddf66-d55qq"]
Sep 16 19:42:38.735759 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:38.735724    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" path="/var/lib/kubelet/pods/e96fabfe-8b34-4c5b-89c5-3b33c15c295d/volumes"
Sep 16 19:42:42.869215 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:42.869179    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:42.927990 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:42:42.927961    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:12.409501 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:12.409461    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:12.510195 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:12.510160    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:12.624728 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:12.624695    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:19.032048 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:19.032014    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:20.266300 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.266262    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"]
Sep 16 19:43:20.267398 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.267372    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main"
Sep 16 19:43:20.267562 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.267549    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main"
Sep 16 19:43:20.267667 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.267657    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="storage-initializer"
Sep 16 19:43:20.268400 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.267746    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="storage-initializer"
Sep 16 19:43:20.268400 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.267770    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d4f629a-0302-4730-ab75-4918d777aa5e" containerName="main"
Sep 16 19:43:20.268400 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.267779    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d4f629a-0302-4730-ab75-4918d777aa5e" containerName="main"
Sep 16 19:43:20.268400 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.267897    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e96fabfe-8b34-4c5b-89c5-3b33c15c295d" containerName="main"
Sep 16 19:43:20.268400 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.267913    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d4f629a-0302-4730-ab75-4918d777aa5e" containerName="main"
Sep 16 19:43:20.271318 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.271299    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.274324 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.274299    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-2rlw2\""
Sep 16 19:43:20.274435 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.274344    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:43:20.274735 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.274714    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 16 19:43:20.274844 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.274748    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 16 19:43:20.275402 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.275384    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 16 19:43:20.275497 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.275406    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 16 19:43:20.282475 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.282452    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"]
Sep 16 19:43:20.356496 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.356426    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:20.400371 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.400336    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-model-cache\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.400371 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.400371    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-home\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.400603 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.400394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljk7l\" (UniqueName: \"kubernetes.io/projected/880e16b1-b963-4be9-b3e6-6ec44e85a55c-kube-api-access-ljk7l\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.400603 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.400522    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880e16b1-b963-4be9-b3e6-6ec44e85a55c-tls-certs\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.400603 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.400583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-dshm\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.400722 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.400613    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/880e16b1-b963-4be9-b3e6-6ec44e85a55c-cabundle-cert\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.400722 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.400635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-tmp-dir\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.400722 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.400677    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.501890 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.501850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-model-cache\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.501890 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.501887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-home\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.502173 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.501906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljk7l\" (UniqueName: \"kubernetes.io/projected/880e16b1-b963-4be9-b3e6-6ec44e85a55c-kube-api-access-ljk7l\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.502173 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.501947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880e16b1-b963-4be9-b3e6-6ec44e85a55c-tls-certs\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.502173 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.501990    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-dshm\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.502173 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.502008    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/880e16b1-b963-4be9-b3e6-6ec44e85a55c-cabundle-cert\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.502173 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.502121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-tmp-dir\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.502426 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.502175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.502426 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.502377    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-model-cache\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.502597 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.502577    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-tmp-dir\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.502648 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.502622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.502695 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.502650    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-home\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.502776 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.502752    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/880e16b1-b963-4be9-b3e6-6ec44e85a55c-cabundle-cert\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.504505 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.504485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-dshm\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.504953 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.504936    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880e16b1-b963-4be9-b3e6-6ec44e85a55c-tls-certs\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.510705 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.510685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljk7l\" (UniqueName: \"kubernetes.io/projected/880e16b1-b963-4be9-b3e6-6ec44e85a55c-kube-api-access-ljk7l\") pod \"lora-single-adapter-test-kserve-645cfbb64-lpg9q\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.582650 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.582538    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:20.884802 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:20.884772    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"]
Sep 16 19:43:20.887114 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:43:20.887083    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880e16b1_b963_4be9_b3e6_6ec44e85a55c.slice/crio-29f6d030ab68a7c4ec72e20dcbc1df5082e04e30b3e01df773cdcedeeca9c8aa WatchSource:0}: Error finding container 29f6d030ab68a7c4ec72e20dcbc1df5082e04e30b3e01df773cdcedeeca9c8aa: Status 404 returned error can't find the container with id 29f6d030ab68a7c4ec72e20dcbc1df5082e04e30b3e01df773cdcedeeca9c8aa
Sep 16 19:43:21.608527 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:21.608486    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" event={"ID":"880e16b1-b963-4be9-b3e6-6ec44e85a55c","Type":"ContainerStarted","Data":"43cbd0b37750762022c1b0835a0eede9ffe5b2064960b3748b989f31e53ca9cf"}
Sep 16 19:43:21.608942 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:21.608575    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" event={"ID":"880e16b1-b963-4be9-b3e6-6ec44e85a55c","Type":"ContainerStarted","Data":"29f6d030ab68a7c4ec72e20dcbc1df5082e04e30b3e01df773cdcedeeca9c8aa"}
Sep 16 19:43:24.627852 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:24.627815    2561 generic.go:356] "Generic (PLEG): container finished" podID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerID="43cbd0b37750762022c1b0835a0eede9ffe5b2064960b3748b989f31e53ca9cf" exitCode=0
Sep 16 19:43:24.628359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:24.627897    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" event={"ID":"880e16b1-b963-4be9-b3e6-6ec44e85a55c","Type":"ContainerDied","Data":"43cbd0b37750762022c1b0835a0eede9ffe5b2064960b3748b989f31e53ca9cf"}
Sep 16 19:43:24.870378 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:24.870314    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:24.936924 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:24.936830    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:25.635733 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:25.635692    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" event={"ID":"880e16b1-b963-4be9-b3e6-6ec44e85a55c","Type":"ContainerStarted","Data":"d6aca4384d3a2e3d7a053b4957113d0b2f62be52c52cb0b0b9f5f9fd9f729a59"}
Sep 16 19:43:25.663371 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:25.663312    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" podStartSLOduration=2.782828267 podStartE2EDuration="5.663292648s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.880464381s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:25.660601525 +0000 UTC m=+3505.463319768" watchObservedRunningTime="2026-09-16 19:43:25.663292648 +0000 UTC m=+3505.466010864"
Sep 16 19:43:26.583129 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:26.583074    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:27.172153 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:27.172103    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:27.532422 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:27.532382    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:30.582845 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:30.582800    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:43:30.584909 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:30.584879    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 19:43:40.582986 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:40.582940    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 19:43:50.010949 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:50.010879    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:50.095557 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:50.095522    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:50.583172 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:43:50.583123    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 19:44:00.582982 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:44:00.582930    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 19:44:10.582880 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:44:10.582830    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 19:44:20.583069 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:44:20.582994    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 19:44:30.583701 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:44:30.583646    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 19:44:40.113737 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:44:40.113702    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:40.226697 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:44:40.226659    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:40.593199 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:44:40.593166    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:44:40.600954 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:44:40.600927    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:44:46.669760 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:44:46.669722    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:51.500877 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:44:51.500744    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:51.755436 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:44:51.755327    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:53.541352 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:44:53.541298    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:53.647775 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:44:53.647737    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:59.353632 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:44:59.353584    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:01.101965 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:01.101933    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:45:01.119394 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:01.119364    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:45:25.846327 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:25.846297    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-645cfbb64-lpg9q_880e16b1-b963-4be9-b3e6-6ec44e85a55c/storage-initializer/0.log"
Sep 16 19:45:25.855553 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:25.855528    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-645cfbb64-lpg9q_880e16b1-b963-4be9-b3e6-6ec44e85a55c/main/0.log"
Sep 16 19:45:27.162331 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:27.162290    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"]
Sep 16 19:45:27.174910 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:27.174876    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:42.185460 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.185422    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerName="main" containerID="cri-o://d6aca4384d3a2e3d7a053b4957113d0b2f62be52c52cb0b0b9f5f9fd9f729a59" gracePeriod=45
Sep 16 19:45:42.594770 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.594724    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 19:45:42.844377 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.844349    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:45:42.923340 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.923263    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-dshm\") pod \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") "
Sep 16 19:45:42.923340 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.923300    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-home\") pod \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") "
Sep 16 19:45:42.923340 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.923335    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-model-cache\") pod \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") "
Sep 16 19:45:42.923603 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.923354    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljk7l\" (UniqueName: \"kubernetes.io/projected/880e16b1-b963-4be9-b3e6-6ec44e85a55c-kube-api-access-ljk7l\") pod \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") "
Sep 16 19:45:42.923603 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.923371    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/880e16b1-b963-4be9-b3e6-6ec44e85a55c-cabundle-cert\") pod \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") "
Sep 16 19:45:42.923603 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.923464    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880e16b1-b963-4be9-b3e6-6ec44e85a55c-tls-certs\") pod \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") "
Sep 16 19:45:42.923603 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.923516    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-tmp-dir\") pod \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") "
Sep 16 19:45:42.923603 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.923555    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-kserve-provision-location\") pod \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\" (UID: \"880e16b1-b963-4be9-b3e6-6ec44e85a55c\") "
Sep 16 19:45:42.923884 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.923621    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-model-cache" (OuterVolumeSpecName: "model-cache") pod "880e16b1-b963-4be9-b3e6-6ec44e85a55c" (UID: "880e16b1-b963-4be9-b3e6-6ec44e85a55c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:42.923884 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.923689    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-home" (OuterVolumeSpecName: "home") pod "880e16b1-b963-4be9-b3e6-6ec44e85a55c" (UID: "880e16b1-b963-4be9-b3e6-6ec44e85a55c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:42.924189 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.923948    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:45:42.924189 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.923968    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:45:42.924189 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.924097    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880e16b1-b963-4be9-b3e6-6ec44e85a55c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "880e16b1-b963-4be9-b3e6-6ec44e85a55c" (UID: "880e16b1-b963-4be9-b3e6-6ec44e85a55c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:45:42.925805 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.925775    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-dshm" (OuterVolumeSpecName: "dshm") pod "880e16b1-b963-4be9-b3e6-6ec44e85a55c" (UID: "880e16b1-b963-4be9-b3e6-6ec44e85a55c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:42.925920 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.925857    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/880e16b1-b963-4be9-b3e6-6ec44e85a55c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "880e16b1-b963-4be9-b3e6-6ec44e85a55c" (UID: "880e16b1-b963-4be9-b3e6-6ec44e85a55c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:45:42.926078 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.926036    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880e16b1-b963-4be9-b3e6-6ec44e85a55c-kube-api-access-ljk7l" (OuterVolumeSpecName: "kube-api-access-ljk7l") pod "880e16b1-b963-4be9-b3e6-6ec44e85a55c" (UID: "880e16b1-b963-4be9-b3e6-6ec44e85a55c"). InnerVolumeSpecName "kube-api-access-ljk7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:45:42.935922 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.935896    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "880e16b1-b963-4be9-b3e6-6ec44e85a55c" (UID: "880e16b1-b963-4be9-b3e6-6ec44e85a55c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:42.947328 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:42.947304    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "880e16b1-b963-4be9-b3e6-6ec44e85a55c" (UID: "880e16b1-b963-4be9-b3e6-6ec44e85a55c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:43.025190 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.025169    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880e16b1-b963-4be9-b3e6-6ec44e85a55c-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:45:43.025190 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.025190    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:45:43.025324 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.025200    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-kserve-provision-location\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:45:43.025324 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.025209    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880e16b1-b963-4be9-b3e6-6ec44e85a55c-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:45:43.025324 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.025219    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljk7l\" (UniqueName: \"kubernetes.io/projected/880e16b1-b963-4be9-b3e6-6ec44e85a55c-kube-api-access-ljk7l\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:45:43.025324 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.025227    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/880e16b1-b963-4be9-b3e6-6ec44e85a55c-cabundle-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:45:43.434671 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.434634    2561 generic.go:356] "Generic (PLEG): container finished" podID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerID="d6aca4384d3a2e3d7a053b4957113d0b2f62be52c52cb0b0b9f5f9fd9f729a59" exitCode=0
Sep 16 19:45:43.435118 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.434693    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"
Sep 16 19:45:43.435118 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.434724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" event={"ID":"880e16b1-b963-4be9-b3e6-6ec44e85a55c","Type":"ContainerDied","Data":"d6aca4384d3a2e3d7a053b4957113d0b2f62be52c52cb0b0b9f5f9fd9f729a59"}
Sep 16 19:45:43.435118 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.434807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q" event={"ID":"880e16b1-b963-4be9-b3e6-6ec44e85a55c","Type":"ContainerDied","Data":"29f6d030ab68a7c4ec72e20dcbc1df5082e04e30b3e01df773cdcedeeca9c8aa"}
Sep 16 19:45:43.435118 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.434831    2561 scope.go:117] "RemoveContainer" containerID="d6aca4384d3a2e3d7a053b4957113d0b2f62be52c52cb0b0b9f5f9fd9f729a59"
Sep 16 19:45:43.444378 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.444360    2561 scope.go:117] "RemoveContainer" containerID="43cbd0b37750762022c1b0835a0eede9ffe5b2064960b3748b989f31e53ca9cf"
Sep 16 19:45:43.453362 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.453336    2561 scope.go:117] "RemoveContainer" containerID="d6aca4384d3a2e3d7a053b4957113d0b2f62be52c52cb0b0b9f5f9fd9f729a59"
Sep 16 19:45:43.453597 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:45:43.453574    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6aca4384d3a2e3d7a053b4957113d0b2f62be52c52cb0b0b9f5f9fd9f729a59\": container with ID starting with d6aca4384d3a2e3d7a053b4957113d0b2f62be52c52cb0b0b9f5f9fd9f729a59 not found: ID does not exist" containerID="d6aca4384d3a2e3d7a053b4957113d0b2f62be52c52cb0b0b9f5f9fd9f729a59"
Sep 16 19:45:43.453658 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.453604    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6aca4384d3a2e3d7a053b4957113d0b2f62be52c52cb0b0b9f5f9fd9f729a59"} err="failed to get container status \"d6aca4384d3a2e3d7a053b4957113d0b2f62be52c52cb0b0b9f5f9fd9f729a59\": rpc error: code = NotFound desc = could not find container \"d6aca4384d3a2e3d7a053b4957113d0b2f62be52c52cb0b0b9f5f9fd9f729a59\": container with ID starting with d6aca4384d3a2e3d7a053b4957113d0b2f62be52c52cb0b0b9f5f9fd9f729a59 not found: ID does not exist"
Sep 16 19:45:43.453658 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.453621    2561 scope.go:117] "RemoveContainer" containerID="43cbd0b37750762022c1b0835a0eede9ffe5b2064960b3748b989f31e53ca9cf"
Sep 16 19:45:43.453854 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:45:43.453837    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43cbd0b37750762022c1b0835a0eede9ffe5b2064960b3748b989f31e53ca9cf\": container with ID starting with 43cbd0b37750762022c1b0835a0eede9ffe5b2064960b3748b989f31e53ca9cf not found: ID does not exist" containerID="43cbd0b37750762022c1b0835a0eede9ffe5b2064960b3748b989f31e53ca9cf"
Sep 16 19:45:43.453904 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.453859    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43cbd0b37750762022c1b0835a0eede9ffe5b2064960b3748b989f31e53ca9cf"} err="failed to get container status \"43cbd0b37750762022c1b0835a0eede9ffe5b2064960b3748b989f31e53ca9cf\": rpc error: code = NotFound desc = could not find container \"43cbd0b37750762022c1b0835a0eede9ffe5b2064960b3748b989f31e53ca9cf\": container with ID starting with 43cbd0b37750762022c1b0835a0eede9ffe5b2064960b3748b989f31e53ca9cf not found: ID does not exist"
Sep 16 19:45:43.459840 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.459818    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"]
Sep 16 19:45:43.465733 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:43.465712    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-645cfbb64-lpg9q"]
Sep 16 19:45:44.735364 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:44.735330    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" path="/var/lib/kubelet/pods/880e16b1-b963-4be9-b3e6-6ec44e85a55c/volumes"
Sep 16 19:45:47.293885 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:47.293853    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:47.421678 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:47.421643    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:47.524827 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:47.524791    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:53.376983 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:53.376945    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:53.478848 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:53.478809    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:54.381527 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.381490    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"]
Sep 16 19:45:54.381935 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.381886    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerName="main"
Sep 16 19:45:54.381935 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.381898    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerName="main"
Sep 16 19:45:54.381935 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.381918    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerName="storage-initializer"
Sep 16 19:45:54.381935 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.381924    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerName="storage-initializer"
Sep 16 19:45:54.382119 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.381986    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="880e16b1-b963-4be9-b3e6-6ec44e85a55c" containerName="main"
Sep 16 19:45:54.385401 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.385379    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.391243 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.391197    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 16 19:45:54.391409 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.391244    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 16 19:45:54.391409 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.391290    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 16 19:45:54.391409 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.391369    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:45:54.391554 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.391500    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 16 19:45:54.392449 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.392346    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-cczzx\""
Sep 16 19:45:54.406662 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.406633    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"]
Sep 16 19:45:54.425217 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.425182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/167355be-3aff-47f6-96fe-57968d983ec5-tls-certs\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.425359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.425230    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.425359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.425291    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/167355be-3aff-47f6-96fe-57968d983ec5-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.425359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.425338    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpj69\" (UniqueName: \"kubernetes.io/projected/167355be-3aff-47f6-96fe-57968d983ec5-kube-api-access-mpj69\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.425552 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.425369    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-home\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.425552 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.425444    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-model-cache\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.425552 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.425484    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-dshm\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.425552 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.425512    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.462493 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.462453    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:54.526586 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.526546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-dshm\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.526770 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.526594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.526770 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.526661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/167355be-3aff-47f6-96fe-57968d983ec5-tls-certs\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.526770 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.526680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.526770 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.526703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/167355be-3aff-47f6-96fe-57968d983ec5-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.526770 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.526738    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpj69\" (UniqueName: \"kubernetes.io/projected/167355be-3aff-47f6-96fe-57968d983ec5-kube-api-access-mpj69\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.526770 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.526756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-home\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.527099 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.526795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-model-cache\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.527197 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.527134    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.527267 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.527245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-model-cache\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.527351 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.527331    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.527631 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.527611    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-home\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.527830 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.527810    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/167355be-3aff-47f6-96fe-57968d983ec5-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.529237 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.529221    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-dshm\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.529520 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.529502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/167355be-3aff-47f6-96fe-57968d983ec5-tls-certs\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.535147 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.535125    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpj69\" (UniqueName: \"kubernetes.io/projected/167355be-3aff-47f6-96fe-57968d983ec5-kube-api-access-mpj69\") pod \"lora-multiple-adapters-test-kserve-8477d56f9-8hx2d\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.696190 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.696102    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:45:54.987264 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:54.987234    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"]
Sep 16 19:45:54.989615 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:45:54.989587    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167355be_3aff_47f6_96fe_57968d983ec5.slice/crio-021fb62ce0e2a20b2f290c948c298d6490eac298bf3cee122eff67b542e40582 WatchSource:0}: Error finding container 021fb62ce0e2a20b2f290c948c298d6490eac298bf3cee122eff67b542e40582: Status 404 returned error can't find the container with id 021fb62ce0e2a20b2f290c948c298d6490eac298bf3cee122eff67b542e40582
Sep 16 19:45:55.503791 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:55.503752    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" event={"ID":"167355be-3aff-47f6-96fe-57968d983ec5","Type":"ContainerStarted","Data":"a184d5c44af1f7dafd268d647aff9ea60102f9549ed6c1d36ec98912adfee800"}
Sep 16 19:45:55.504225 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:55.503817    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" event={"ID":"167355be-3aff-47f6-96fe-57968d983ec5","Type":"ContainerStarted","Data":"021fb62ce0e2a20b2f290c948c298d6490eac298bf3cee122eff67b542e40582"}
Sep 16 19:45:58.520864 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:58.520829    2561 generic.go:356] "Generic (PLEG): container finished" podID="167355be-3aff-47f6-96fe-57968d983ec5" containerID="a184d5c44af1f7dafd268d647aff9ea60102f9549ed6c1d36ec98912adfee800" exitCode=0
Sep 16 19:45:58.521273 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:58.520902    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" event={"ID":"167355be-3aff-47f6-96fe-57968d983ec5","Type":"ContainerDied","Data":"a184d5c44af1f7dafd268d647aff9ea60102f9549ed6c1d36ec98912adfee800"}
Sep 16 19:45:59.530205 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:59.530164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" event={"ID":"167355be-3aff-47f6-96fe-57968d983ec5","Type":"ContainerStarted","Data":"22c24b3ed569c1ba9a4748d66513e8f725833c3c3cc7b57ac10b69d8e34b9855"}
Sep 16 19:45:59.557750 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:59.557687    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" podStartSLOduration=2.517030596 podStartE2EDuration="5.557665641s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.040635045s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:45:59.555180378 +0000 UTC m=+3659.357898596" watchObservedRunningTime="2026-09-16 19:45:59.557665641 +0000 UTC m=+3659.360383858"
Sep 16 19:45:59.697001 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:45:59.696965    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:46:04.696950 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:46:04.696908    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:46:04.698710 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:46:04.698679    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" podUID="167355be-3aff-47f6-96fe-57968d983ec5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:46:14.696840 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:46:14.696798    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" podUID="167355be-3aff-47f6-96fe-57968d983ec5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:46:24.696671 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:46:24.696577    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" podUID="167355be-3aff-47f6-96fe-57968d983ec5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:46:34.697077 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:46:34.697017    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" podUID="167355be-3aff-47f6-96fe-57968d983ec5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:46:44.697470 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:46:44.697424    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" podUID="167355be-3aff-47f6-96fe-57968d983ec5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:46:48.905473 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:46:48.905440    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:54.697584 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:46:54.697532    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" podUID="167355be-3aff-47f6-96fe-57968d983ec5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:46:56.729684 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:46:56.729653    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:56.801953 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:46:56.801912    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:04.697568 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:04.697524    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" podUID="167355be-3aff-47f6-96fe-57968d983ec5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:47:14.707041 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:14.707002    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:47:14.714848 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:14.714820    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:47:41.029049 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:41.029015    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-8477d56f9-8hx2d_167355be-3aff-47f6-96fe-57968d983ec5/storage-initializer/0.log"
Sep 16 19:47:41.039424 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:41.039398    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-8477d56f9-8hx2d_167355be-3aff-47f6-96fe-57968d983ec5/main/0.log"
Sep 16 19:47:41.975491 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:41.975452    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"]
Sep 16 19:47:42.003241 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:42.003207    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:46.825435 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:46.825393    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:46.924300 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:46.924271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:47.161720 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:47.161677    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:47.296106 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:47.296037    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:51.015447 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:51.015415    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:52.206785 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:52.206750    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:52.456357 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:52.456315    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:52.557817 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:52.557785    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:54.108378 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:54.108342    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:56.999230 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:56.999164    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" podUID="167355be-3aff-47f6-96fe-57968d983ec5" containerName="main" containerID="cri-o://22c24b3ed569c1ba9a4748d66513e8f725833c3c3cc7b57ac10b69d8e34b9855" gracePeriod=45
Sep 16 19:47:57.648388 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.648360    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:47:57.705238 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.705157    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-tmp-dir\") pod \"167355be-3aff-47f6-96fe-57968d983ec5\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") "
Sep 16 19:47:57.705238 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.705208    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpj69\" (UniqueName: \"kubernetes.io/projected/167355be-3aff-47f6-96fe-57968d983ec5-kube-api-access-mpj69\") pod \"167355be-3aff-47f6-96fe-57968d983ec5\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") "
Sep 16 19:47:57.705238 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.705235    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-model-cache\") pod \"167355be-3aff-47f6-96fe-57968d983ec5\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") "
Sep 16 19:47:57.705491 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.705307    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-dshm\") pod \"167355be-3aff-47f6-96fe-57968d983ec5\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") "
Sep 16 19:47:57.705491 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.705337    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-kserve-provision-location\") pod \"167355be-3aff-47f6-96fe-57968d983ec5\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") "
Sep 16 19:47:57.705491 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.705412    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-home\") pod \"167355be-3aff-47f6-96fe-57968d983ec5\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") "
Sep 16 19:47:57.705491 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.705486    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/167355be-3aff-47f6-96fe-57968d983ec5-cabundle-cert\") pod \"167355be-3aff-47f6-96fe-57968d983ec5\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") "
Sep 16 19:47:57.705712 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.705538    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-model-cache" (OuterVolumeSpecName: "model-cache") pod "167355be-3aff-47f6-96fe-57968d983ec5" (UID: "167355be-3aff-47f6-96fe-57968d983ec5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:57.705851 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.705809    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-home" (OuterVolumeSpecName: "home") pod "167355be-3aff-47f6-96fe-57968d983ec5" (UID: "167355be-3aff-47f6-96fe-57968d983ec5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:57.705982 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.705868    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/167355be-3aff-47f6-96fe-57968d983ec5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "167355be-3aff-47f6-96fe-57968d983ec5" (UID: "167355be-3aff-47f6-96fe-57968d983ec5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:47:57.705982 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.705516    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/167355be-3aff-47f6-96fe-57968d983ec5-tls-certs\") pod \"167355be-3aff-47f6-96fe-57968d983ec5\" (UID: \"167355be-3aff-47f6-96fe-57968d983ec5\") "
Sep 16 19:47:57.706302 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.706243    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:47:57.706302 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.706270    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/167355be-3aff-47f6-96fe-57968d983ec5-cabundle-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:47:57.706302 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.706286    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:47:57.708033 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.707980    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167355be-3aff-47f6-96fe-57968d983ec5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "167355be-3aff-47f6-96fe-57968d983ec5" (UID: "167355be-3aff-47f6-96fe-57968d983ec5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:47:57.708272 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.708244    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167355be-3aff-47f6-96fe-57968d983ec5-kube-api-access-mpj69" (OuterVolumeSpecName: "kube-api-access-mpj69") pod "167355be-3aff-47f6-96fe-57968d983ec5" (UID: "167355be-3aff-47f6-96fe-57968d983ec5"). InnerVolumeSpecName "kube-api-access-mpj69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:47:57.708272 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.708245    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-dshm" (OuterVolumeSpecName: "dshm") pod "167355be-3aff-47f6-96fe-57968d983ec5" (UID: "167355be-3aff-47f6-96fe-57968d983ec5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:57.717268 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.717245    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "167355be-3aff-47f6-96fe-57968d983ec5" (UID: "167355be-3aff-47f6-96fe-57968d983ec5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:57.730804 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.730781    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "167355be-3aff-47f6-96fe-57968d983ec5" (UID: "167355be-3aff-47f6-96fe-57968d983ec5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:57.807466 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.807423    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:47:57.807466 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.807468    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-kserve-provision-location\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:47:57.807665 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.807483    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/167355be-3aff-47f6-96fe-57968d983ec5-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:47:57.807665 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.807496    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167355be-3aff-47f6-96fe-57968d983ec5-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:47:57.807665 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:57.807510    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpj69\" (UniqueName: \"kubernetes.io/projected/167355be-3aff-47f6-96fe-57968d983ec5-kube-api-access-mpj69\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:47:58.212439 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:58.212397    2561 generic.go:356] "Generic (PLEG): container finished" podID="167355be-3aff-47f6-96fe-57968d983ec5" containerID="22c24b3ed569c1ba9a4748d66513e8f725833c3c3cc7b57ac10b69d8e34b9855" exitCode=0
Sep 16 19:47:58.213023 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:58.212464    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"
Sep 16 19:47:58.213023 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:58.212477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" event={"ID":"167355be-3aff-47f6-96fe-57968d983ec5","Type":"ContainerDied","Data":"22c24b3ed569c1ba9a4748d66513e8f725833c3c3cc7b57ac10b69d8e34b9855"}
Sep 16 19:47:58.213023 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:58.212530    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d" event={"ID":"167355be-3aff-47f6-96fe-57968d983ec5","Type":"ContainerDied","Data":"021fb62ce0e2a20b2f290c948c298d6490eac298bf3cee122eff67b542e40582"}
Sep 16 19:47:58.213023 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:58.212559    2561 scope.go:117] "RemoveContainer" containerID="22c24b3ed569c1ba9a4748d66513e8f725833c3c3cc7b57ac10b69d8e34b9855"
Sep 16 19:47:58.222974 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:58.222953    2561 scope.go:117] "RemoveContainer" containerID="a184d5c44af1f7dafd268d647aff9ea60102f9549ed6c1d36ec98912adfee800"
Sep 16 19:47:58.231587 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:58.231568    2561 scope.go:117] "RemoveContainer" containerID="22c24b3ed569c1ba9a4748d66513e8f725833c3c3cc7b57ac10b69d8e34b9855"
Sep 16 19:47:58.231870 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:47:58.231850    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c24b3ed569c1ba9a4748d66513e8f725833c3c3cc7b57ac10b69d8e34b9855\": container with ID starting with 22c24b3ed569c1ba9a4748d66513e8f725833c3c3cc7b57ac10b69d8e34b9855 not found: ID does not exist" containerID="22c24b3ed569c1ba9a4748d66513e8f725833c3c3cc7b57ac10b69d8e34b9855"
Sep 16 19:47:58.231918 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:58.231877    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c24b3ed569c1ba9a4748d66513e8f725833c3c3cc7b57ac10b69d8e34b9855"} err="failed to get container status \"22c24b3ed569c1ba9a4748d66513e8f725833c3c3cc7b57ac10b69d8e34b9855\": rpc error: code = NotFound desc = could not find container \"22c24b3ed569c1ba9a4748d66513e8f725833c3c3cc7b57ac10b69d8e34b9855\": container with ID starting with 22c24b3ed569c1ba9a4748d66513e8f725833c3c3cc7b57ac10b69d8e34b9855 not found: ID does not exist"
Sep 16 19:47:58.231918 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:58.231895    2561 scope.go:117] "RemoveContainer" containerID="a184d5c44af1f7dafd268d647aff9ea60102f9549ed6c1d36ec98912adfee800"
Sep 16 19:47:58.232183 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:47:58.232168    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a184d5c44af1f7dafd268d647aff9ea60102f9549ed6c1d36ec98912adfee800\": container with ID starting with a184d5c44af1f7dafd268d647aff9ea60102f9549ed6c1d36ec98912adfee800 not found: ID does not exist" containerID="a184d5c44af1f7dafd268d647aff9ea60102f9549ed6c1d36ec98912adfee800"
Sep 16 19:47:58.232230 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:58.232187    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a184d5c44af1f7dafd268d647aff9ea60102f9549ed6c1d36ec98912adfee800"} err="failed to get container status \"a184d5c44af1f7dafd268d647aff9ea60102f9549ed6c1d36ec98912adfee800\": rpc error: code = NotFound desc = could not find container \"a184d5c44af1f7dafd268d647aff9ea60102f9549ed6c1d36ec98912adfee800\": container with ID starting with a184d5c44af1f7dafd268d647aff9ea60102f9549ed6c1d36ec98912adfee800 not found: ID does not exist"
Sep 16 19:47:58.240343 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:58.240318    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"]
Sep 16 19:47:58.244900 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:58.244862    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8477d56f9-8hx2d"]
Sep 16 19:47:58.734757 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:47:58.734728    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="167355be-3aff-47f6-96fe-57968d983ec5" path="/var/lib/kubelet/pods/167355be-3aff-47f6-96fe-57968d983ec5/volumes"
Sep 16 19:48:00.088825 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:00.088762    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:00.172467 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:00.172432    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:02.171404 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:02.171365    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:02.275486 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:02.275455    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:02.377261 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:02.377230    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:02.457218 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:02.457135    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:02.529030 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:02.529002    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:05.070193 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.070160    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:05.151011 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.150978    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:05.472558 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.472528    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"]
Sep 16 19:48:05.472911 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.472899    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="167355be-3aff-47f6-96fe-57968d983ec5" containerName="storage-initializer"
Sep 16 19:48:05.472957 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.472913    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="167355be-3aff-47f6-96fe-57968d983ec5" containerName="storage-initializer"
Sep 16 19:48:05.472957 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.472934    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="167355be-3aff-47f6-96fe-57968d983ec5" containerName="main"
Sep 16 19:48:05.472957 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.472939    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="167355be-3aff-47f6-96fe-57968d983ec5" containerName="main"
Sep 16 19:48:05.473052 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.473004    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="167355be-3aff-47f6-96fe-57968d983ec5" containerName="main"
Sep 16 19:48:05.477603 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.477586    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"
Sep 16 19:48:05.480679 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.480660    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 16 19:48:05.480845 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.480829    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-zpm2m\""
Sep 16 19:48:05.481726 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.481710    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 16 19:48:05.481816 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.481734    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 16 19:48:05.485536 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.485514    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"]
Sep 16 19:48:05.575960 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.575928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ffr\" (UniqueName: \"kubernetes.io/projected/9d652046-92b0-4364-8463-d3ea14240a92-kube-api-access-m7ffr\") pod \"e2e-pvc-lora-download-bq8q4\" (UID: \"9d652046-92b0-4364-8463-d3ea14240a92\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"
Sep 16 19:48:05.588363 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.588336    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:05.677052 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.677025    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ffr\" (UniqueName: \"kubernetes.io/projected/9d652046-92b0-4364-8463-d3ea14240a92-kube-api-access-m7ffr\") pod \"e2e-pvc-lora-download-bq8q4\" (UID: \"9d652046-92b0-4364-8463-d3ea14240a92\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"
Sep 16 19:48:05.687528 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:05.687500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ffr\" (UniqueName: \"kubernetes.io/projected/9d652046-92b0-4364-8463-d3ea14240a92-kube-api-access-m7ffr\") pod \"e2e-pvc-lora-download-bq8q4\" (UID: \"9d652046-92b0-4364-8463-d3ea14240a92\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"
Sep 16 19:48:07.896088 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:07.896026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"e2e-pvc-lora-download-bq8q4\" (UID: \"9d652046-92b0-4364-8463-d3ea14240a92\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"
Sep 16 19:48:07.902210 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:07.902187    2561 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"e2e-pvc-lora-download-bq8q4\" (UID: \"9d652046-92b0-4364-8463-d3ea14240a92\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"
Sep 16 19:48:07.997445 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:07.997417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"e2e-pvc-lora-download-bq8q4\" (UID: \"9d652046-92b0-4364-8463-d3ea14240a92\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"
Sep 16 19:48:07.997584 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:07.997540    2561 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"e2e-pvc-lora-download-bq8q4\" (UID: \"9d652046-92b0-4364-8463-d3ea14240a92\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"
Sep 16 19:48:08.001196 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:08.001172    2561 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"e2e-pvc-lora-download-bq8q4\" (UID: \"9d652046-92b0-4364-8463-d3ea14240a92\") DevicePath \"csi-c74ba77b2b7de16d3941a6d0570a96a36c3a0ec5d58a0311e247137126d13b9d\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"
Sep 16 19:48:08.224513 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:08.224473    2561 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"e2e-pvc-lora-download-bq8q4\" (UID: \"9d652046-92b0-4364-8463-d3ea14240a92\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3ac8997ae6bc15125020c24338fefd2bf7a168915614466bd342104ece7ac7b3/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"
Sep 16 19:48:08.238679 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:08.238651    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"e2e-pvc-lora-download-bq8q4\" (UID: \"9d652046-92b0-4364-8463-d3ea14240a92\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"
Sep 16 19:48:08.487873 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:08.487786    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"
Sep 16 19:48:08.624991 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:08.624943    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:08.727234 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:08.727178    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:08.762652 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:08.762626    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"]
Sep 16 19:48:08.763650 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:48:08.763625    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d652046_92b0_4364_8463_d3ea14240a92.slice/crio-840b199925b159a6a3c5798c54ee8c933de5e4f84437b8c23b967ed13b47e86c WatchSource:0}: Error finding container 840b199925b159a6a3c5798c54ee8c933de5e4f84437b8c23b967ed13b47e86c: Status 404 returned error can't find the container with id 840b199925b159a6a3c5798c54ee8c933de5e4f84437b8c23b967ed13b47e86c
Sep 16 19:48:09.273672 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:09.273635    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4" event={"ID":"9d652046-92b0-4364-8463-d3ea14240a92","Type":"ContainerStarted","Data":"53c95d3535af5b311a57c165480f399efb4474a8fce1721d49bca09e10f83dd5"}
Sep 16 19:48:09.274039 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:09.273680    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4" event={"ID":"9d652046-92b0-4364-8463-d3ea14240a92","Type":"ContainerStarted","Data":"840b199925b159a6a3c5798c54ee8c933de5e4f84437b8c23b967ed13b47e86c"}
Sep 16 19:48:11.285949 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:11.285907    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d652046-92b0-4364-8463-d3ea14240a92" containerID="53c95d3535af5b311a57c165480f399efb4474a8fce1721d49bca09e10f83dd5" exitCode=0
Sep 16 19:48:11.286370 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:11.285978    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4" event={"ID":"9d652046-92b0-4364-8463-d3ea14240a92","Type":"ContainerDied","Data":"53c95d3535af5b311a57c165480f399efb4474a8fce1721d49bca09e10f83dd5"}
Sep 16 19:48:12.431544 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:12.431514    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"
Sep 16 19:48:12.435545 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:12.435528    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"9d652046-92b0-4364-8463-d3ea14240a92\" (UID: \"9d652046-92b0-4364-8463-d3ea14240a92\") "
Sep 16 19:48:12.435606 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:12.435589    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7ffr\" (UniqueName: \"kubernetes.io/projected/9d652046-92b0-4364-8463-d3ea14240a92-kube-api-access-m7ffr\") pod \"9d652046-92b0-4364-8463-d3ea14240a92\" (UID: \"9d652046-92b0-4364-8463-d3ea14240a92\") "
Sep 16 19:48:12.438165 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:12.438133    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d652046-92b0-4364-8463-d3ea14240a92-kube-api-access-m7ffr" (OuterVolumeSpecName: "kube-api-access-m7ffr") pod "9d652046-92b0-4364-8463-d3ea14240a92" (UID: "9d652046-92b0-4364-8463-d3ea14240a92"). InnerVolumeSpecName "kube-api-access-m7ffr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:48:12.439745 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:12.439716    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec" (OuterVolumeSpecName: "model-storage") pod "9d652046-92b0-4364-8463-d3ea14240a92" (UID: "9d652046-92b0-4364-8463-d3ea14240a92"). InnerVolumeSpecName "pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:48:12.536208 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:12.536176    2561 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") on node \"ip-10-0-129-125.ec2.internal\" "
Sep 16 19:48:12.536208 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:12.536200    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7ffr\" (UniqueName: \"kubernetes.io/projected/9d652046-92b0-4364-8463-d3ea14240a92-kube-api-access-m7ffr\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:48:13.297954 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:13.297928    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"
Sep 16 19:48:13.298162 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:13.297923    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4" event={"ID":"9d652046-92b0-4364-8463-d3ea14240a92","Type":"ContainerDied","Data":"840b199925b159a6a3c5798c54ee8c933de5e4f84437b8c23b967ed13b47e86c"}
Sep 16 19:48:13.298162 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:13.298042    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="840b199925b159a6a3c5798c54ee8c933de5e4f84437b8c23b967ed13b47e86c"
Sep 16 19:48:13.507241 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:13.507213    2561 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec") on node "ip-10-0-129-125.ec2.internal"
Sep 16 19:48:13.548957 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:13.543373    2561 reconciler_common.go:299] "Volume detached for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"csi-c74ba77b2b7de16d3941a6d0570a96a36c3a0ec5d58a0311e247137126d13b9d\""
Sep 16 19:48:32.064913 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.064858    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"]
Sep 16 19:48:32.065591 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.065559    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d652046-92b0-4364-8463-d3ea14240a92" containerName="storage-initializer"
Sep 16 19:48:32.065591 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.065583    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d652046-92b0-4364-8463-d3ea14240a92" containerName="storage-initializer"
Sep 16 19:48:32.065697 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.065685    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d652046-92b0-4364-8463-d3ea14240a92" containerName="storage-initializer"
Sep 16 19:48:32.068870 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.068853    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.071341 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.071312    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 16 19:48:32.071570 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.071551    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-zpm2m\""
Sep 16 19:48:32.071711 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.071692    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 16 19:48:32.071850 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.071830    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 16 19:48:32.072596 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.072574    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:48:32.072712 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.072629    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 16 19:48:32.085129 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.085073    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"]
Sep 16 19:48:32.094369 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.094341    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:32.103299 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.103268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-home\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.103453 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.103309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-dshm\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.103453 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.103411    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn25r\" (UniqueName: \"kubernetes.io/projected/79811896-54dd-4f30-9a37-f807f74dcd23-kube-api-access-xn25r\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.103536 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.103492    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79811896-54dd-4f30-9a37-f807f74dcd23-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.103580 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.103559    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.103617 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.103602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.103663 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.103645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.103711 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.103687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79811896-54dd-4f30-9a37-f807f74dcd23-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.204623 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.204583    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79811896-54dd-4f30-9a37-f807f74dcd23-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.204836 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.204651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-home\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.204836 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.204679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-dshm\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.204964 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.204907    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn25r\" (UniqueName: \"kubernetes.io/projected/79811896-54dd-4f30-9a37-f807f74dcd23-kube-api-access-xn25r\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.205025 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.204957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79811896-54dd-4f30-9a37-f807f74dcd23-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.205214 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.205030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.205214 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.205096    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-home\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.205214 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.205140    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.205457 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.205434    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79811896-54dd-4f30-9a37-f807f74dcd23-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.205524 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.205498    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.205907 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.205629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.205907 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.205698    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.205907 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.205830    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.207782 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.207757    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-dshm\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.208140 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.208120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79811896-54dd-4f30-9a37-f807f74dcd23-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:32.213889 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:32.213866    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn25r\" (UniqueName: \"kubernetes.io/projected/79811896-54dd-4f30-9a37-f807f74dcd23-kube-api-access-xn25r\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:36.542188 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:36.542152    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:36.549925 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:36.549899    2561 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:36.643452 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:36.643422    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:36.643633 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:36.643558    2561 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:36.647187 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:36.647169    2561 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") DevicePath \"csi-c74ba77b2b7de16d3941a6d0570a96a36c3a0ec5d58a0311e247137126d13b9d\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:36.739473 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:36.739445    2561 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3ac8997ae6bc15125020c24338fefd2bf7a168915614466bd342104ece7ac7b3/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:36.758596 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:36.758571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:36.881444 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:36.881368    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:37.166189 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:37.166161    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"]
Sep 16 19:48:37.167397 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:48:37.167367    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79811896_54dd_4f30_9a37_f807f74dcd23.slice/crio-37406bc9f69d24298f68b58a2e21ed636df24e0545b7c1bda03ec6f040e1e376 WatchSource:0}: Error finding container 37406bc9f69d24298f68b58a2e21ed636df24e0545b7c1bda03ec6f040e1e376: Status 404 returned error can't find the container with id 37406bc9f69d24298f68b58a2e21ed636df24e0545b7c1bda03ec6f040e1e376
Sep 16 19:48:37.433848 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:37.433742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" event={"ID":"79811896-54dd-4f30-9a37-f807f74dcd23","Type":"ContainerStarted","Data":"cc3a2e2b21228026f1cf00cc244b32b7b431709022128140f7f813de6ed2664f"}
Sep 16 19:48:37.433848 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:37.433823    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" event={"ID":"79811896-54dd-4f30-9a37-f807f74dcd23","Type":"ContainerStarted","Data":"37406bc9f69d24298f68b58a2e21ed636df24e0545b7c1bda03ec6f040e1e376"}
Sep 16 19:48:40.453807 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:40.453772    2561 generic.go:356] "Generic (PLEG): container finished" podID="79811896-54dd-4f30-9a37-f807f74dcd23" containerID="cc3a2e2b21228026f1cf00cc244b32b7b431709022128140f7f813de6ed2664f" exitCode=0
Sep 16 19:48:40.454284 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:40.453845    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" event={"ID":"79811896-54dd-4f30-9a37-f807f74dcd23","Type":"ContainerDied","Data":"cc3a2e2b21228026f1cf00cc244b32b7b431709022128140f7f813de6ed2664f"}
Sep 16 19:48:41.461223 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:41.461187    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" event={"ID":"79811896-54dd-4f30-9a37-f807f74dcd23","Type":"ContainerStarted","Data":"663bb2326d9e2c98b2918f89ceb51b72904cf2656d837309263e7af5d9f7b78a"}
Sep 16 19:48:41.488819 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:41.488776    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" podStartSLOduration=7.268802855 podStartE2EDuration="9.488760934s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.219958079s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:48:41.484904782 +0000 UTC m=+3821.287623011" watchObservedRunningTime="2026-09-16 19:48:41.488760934 +0000 UTC m=+3821.291479149"
Sep 16 19:48:41.881603 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:41.881558    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:46.882604 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:46.882564    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:48:46.884467 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:46.884436    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" podUID="79811896-54dd-4f30-9a37-f807f74dcd23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:48:56.882777 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:48:56.882728    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" podUID="79811896-54dd-4f30-9a37-f807f74dcd23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:49:06.882530 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:49:06.882474    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" podUID="79811896-54dd-4f30-9a37-f807f74dcd23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:49:16.882654 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:49:16.882608    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" podUID="79811896-54dd-4f30-9a37-f807f74dcd23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:49:26.882365 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:49:26.882281    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" podUID="79811896-54dd-4f30-9a37-f807f74dcd23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:49:36.389325 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:49:36.389277    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:36.490775 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:49:36.490741    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:36.882552 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:49:36.882501    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" podUID="79811896-54dd-4f30-9a37-f807f74dcd23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:49:46.882701 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:49:46.882653    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" podUID="79811896-54dd-4f30-9a37-f807f74dcd23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:49:56.892779 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:49:56.892744    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:49:56.900446 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:49:56.900415    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:49:57.586936 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:49:57.586897    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:57.709351 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:49:57.709314    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:01.151402 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:01.151369    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:50:01.167947 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:01.167925    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:50:21.635039 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:21.635007    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v_79811896-54dd-4f30-9a37-f807f74dcd23/storage-initializer/0.log"
Sep 16 19:50:21.644101 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:21.644076    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v_79811896-54dd-4f30-9a37-f807f74dcd23/main/0.log"
Sep 16 19:50:22.965180 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:22.965145    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"]
Sep 16 19:50:23.003463 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:23.003365    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:30.037752 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:30.037720    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:37.990270 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:37.990213    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" podUID="79811896-54dd-4f30-9a37-f807f74dcd23" containerName="main" containerID="cri-o://663bb2326d9e2c98b2918f89ceb51b72904cf2656d837309263e7af5d9f7b78a" gracePeriod=45
Sep 16 19:50:38.652110 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.652082    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:50:38.800907 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.800877    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-model-cache\") pod \"79811896-54dd-4f30-9a37-f807f74dcd23\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") "
Sep 16 19:50:38.801090 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.800976    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") pod \"79811896-54dd-4f30-9a37-f807f74dcd23\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") "
Sep 16 19:50:38.801090 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.801025    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79811896-54dd-4f30-9a37-f807f74dcd23-tls-certs\") pod \"79811896-54dd-4f30-9a37-f807f74dcd23\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") "
Sep 16 19:50:38.801090 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.801049    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-home\") pod \"79811896-54dd-4f30-9a37-f807f74dcd23\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") "
Sep 16 19:50:38.801251 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.801111    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-kserve-provision-location\") pod \"79811896-54dd-4f30-9a37-f807f74dcd23\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") "
Sep 16 19:50:38.801251 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.801163    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-dshm\") pod \"79811896-54dd-4f30-9a37-f807f74dcd23\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") "
Sep 16 19:50:38.801251 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.801181    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xn25r\" (UniqueName: \"kubernetes.io/projected/79811896-54dd-4f30-9a37-f807f74dcd23-kube-api-access-xn25r\") pod \"79811896-54dd-4f30-9a37-f807f74dcd23\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") "
Sep 16 19:50:38.801251 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.801186    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-model-cache" (OuterVolumeSpecName: "model-cache") pod "79811896-54dd-4f30-9a37-f807f74dcd23" (UID: "79811896-54dd-4f30-9a37-f807f74dcd23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:38.801251 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.801220    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-tmp-dir\") pod \"79811896-54dd-4f30-9a37-f807f74dcd23\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") "
Sep 16 19:50:38.801251 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.801243    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79811896-54dd-4f30-9a37-f807f74dcd23-cabundle-cert\") pod \"79811896-54dd-4f30-9a37-f807f74dcd23\" (UID: \"79811896-54dd-4f30-9a37-f807f74dcd23\") "
Sep 16 19:50:38.801575 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.801547    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:50:38.801636 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.801603    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-home" (OuterVolumeSpecName: "home") pod "79811896-54dd-4f30-9a37-f807f74dcd23" (UID: "79811896-54dd-4f30-9a37-f807f74dcd23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:38.801970 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.801946    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79811896-54dd-4f30-9a37-f807f74dcd23-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79811896-54dd-4f30-9a37-f807f74dcd23" (UID: "79811896-54dd-4f30-9a37-f807f74dcd23"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:50:38.803599 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.803563    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-dshm" (OuterVolumeSpecName: "dshm") pod "79811896-54dd-4f30-9a37-f807f74dcd23" (UID: "79811896-54dd-4f30-9a37-f807f74dcd23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:38.804032 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.803819    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79811896-54dd-4f30-9a37-f807f74dcd23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "79811896-54dd-4f30-9a37-f807f74dcd23" (UID: "79811896-54dd-4f30-9a37-f807f74dcd23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:50:38.804032 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.803926    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79811896-54dd-4f30-9a37-f807f74dcd23-kube-api-access-xn25r" (OuterVolumeSpecName: "kube-api-access-xn25r") pod "79811896-54dd-4f30-9a37-f807f74dcd23" (UID: "79811896-54dd-4f30-9a37-f807f74dcd23"). InnerVolumeSpecName "kube-api-access-xn25r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:50:38.806630 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.806605    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "79811896-54dd-4f30-9a37-f807f74dcd23" (UID: "79811896-54dd-4f30-9a37-f807f74dcd23"). InnerVolumeSpecName "pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:50:38.813858 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.813834    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "79811896-54dd-4f30-9a37-f807f74dcd23" (UID: "79811896-54dd-4f30-9a37-f807f74dcd23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:38.826539 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.826514    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79811896-54dd-4f30-9a37-f807f74dcd23" (UID: "79811896-54dd-4f30-9a37-f807f74dcd23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:38.902307 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.902281    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-kserve-provision-location\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:50:38.902307 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.902307    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:50:38.902471 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.902318    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn25r\" (UniqueName: \"kubernetes.io/projected/79811896-54dd-4f30-9a37-f807f74dcd23-kube-api-access-xn25r\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:50:38.902471 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.902328    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:50:38.902471 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.902336    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79811896-54dd-4f30-9a37-f807f74dcd23-cabundle-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:50:38.902471 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.902361    2561 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") on node \"ip-10-0-129-125.ec2.internal\" "
Sep 16 19:50:38.902471 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.902369    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79811896-54dd-4f30-9a37-f807f74dcd23-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:50:38.902471 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.902378    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79811896-54dd-4f30-9a37-f807f74dcd23-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:50:38.912751 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:38.912730    2561 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec") on node "ip-10-0-129-125.ec2.internal"
Sep 16 19:50:39.003097 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:39.003042    2561 reconciler_common.go:299] "Volume detached for volume \"pvc-2cc1bcbd-3bba-4a6f-a377-71ce832d71b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-071e6d0aa5359feec\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"csi-c74ba77b2b7de16d3941a6d0570a96a36c3a0ec5d58a0311e247137126d13b9d\""
Sep 16 19:50:39.173317 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:39.173230    2561 generic.go:356] "Generic (PLEG): container finished" podID="79811896-54dd-4f30-9a37-f807f74dcd23" containerID="663bb2326d9e2c98b2918f89ceb51b72904cf2656d837309263e7af5d9f7b78a" exitCode=0
Sep 16 19:50:39.173317 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:39.173280    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"
Sep 16 19:50:39.173317 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:39.173300    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" event={"ID":"79811896-54dd-4f30-9a37-f807f74dcd23","Type":"ContainerDied","Data":"663bb2326d9e2c98b2918f89ceb51b72904cf2656d837309263e7af5d9f7b78a"}
Sep 16 19:50:39.173519 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:39.173349    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v" event={"ID":"79811896-54dd-4f30-9a37-f807f74dcd23","Type":"ContainerDied","Data":"37406bc9f69d24298f68b58a2e21ed636df24e0545b7c1bda03ec6f040e1e376"}
Sep 16 19:50:39.173519 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:39.173373    2561 scope.go:117] "RemoveContainer" containerID="663bb2326d9e2c98b2918f89ceb51b72904cf2656d837309263e7af5d9f7b78a"
Sep 16 19:50:39.183857 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:39.183838    2561 scope.go:117] "RemoveContainer" containerID="cc3a2e2b21228026f1cf00cc244b32b7b431709022128140f7f813de6ed2664f"
Sep 16 19:50:39.192464 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:39.192444    2561 scope.go:117] "RemoveContainer" containerID="663bb2326d9e2c98b2918f89ceb51b72904cf2656d837309263e7af5d9f7b78a"
Sep 16 19:50:39.192743 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:50:39.192725    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663bb2326d9e2c98b2918f89ceb51b72904cf2656d837309263e7af5d9f7b78a\": container with ID starting with 663bb2326d9e2c98b2918f89ceb51b72904cf2656d837309263e7af5d9f7b78a not found: ID does not exist" containerID="663bb2326d9e2c98b2918f89ceb51b72904cf2656d837309263e7af5d9f7b78a"
Sep 16 19:50:39.192791 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:39.192751    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663bb2326d9e2c98b2918f89ceb51b72904cf2656d837309263e7af5d9f7b78a"} err="failed to get container status \"663bb2326d9e2c98b2918f89ceb51b72904cf2656d837309263e7af5d9f7b78a\": rpc error: code = NotFound desc = could not find container \"663bb2326d9e2c98b2918f89ceb51b72904cf2656d837309263e7af5d9f7b78a\": container with ID starting with 663bb2326d9e2c98b2918f89ceb51b72904cf2656d837309263e7af5d9f7b78a not found: ID does not exist"
Sep 16 19:50:39.192791 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:39.192771    2561 scope.go:117] "RemoveContainer" containerID="cc3a2e2b21228026f1cf00cc244b32b7b431709022128140f7f813de6ed2664f"
Sep 16 19:50:39.193007 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:50:39.192992    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3a2e2b21228026f1cf00cc244b32b7b431709022128140f7f813de6ed2664f\": container with ID starting with cc3a2e2b21228026f1cf00cc244b32b7b431709022128140f7f813de6ed2664f not found: ID does not exist" containerID="cc3a2e2b21228026f1cf00cc244b32b7b431709022128140f7f813de6ed2664f"
Sep 16 19:50:39.193048 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:39.193010    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3a2e2b21228026f1cf00cc244b32b7b431709022128140f7f813de6ed2664f"} err="failed to get container status \"cc3a2e2b21228026f1cf00cc244b32b7b431709022128140f7f813de6ed2664f\": rpc error: code = NotFound desc = could not find container \"cc3a2e2b21228026f1cf00cc244b32b7b431709022128140f7f813de6ed2664f\": container with ID starting with cc3a2e2b21228026f1cf00cc244b32b7b431709022128140f7f813de6ed2664f not found: ID does not exist"
Sep 16 19:50:39.198151 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:39.198121    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"]
Sep 16 19:50:39.202791 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:39.202769    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-665bc7c7d-92p9v"]
Sep 16 19:50:40.734623 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:40.734588    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79811896-54dd-4f30-9a37-f807f74dcd23" path="/var/lib/kubelet/pods/79811896-54dd-4f30-9a37-f807f74dcd23/volumes"
Sep 16 19:50:42.879357 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:42.879299    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:42.986901 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:42.986873    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:43.079245 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:43.079203    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:43.181340 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:43.181225    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:43.259968 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:43.259919    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:46.139121 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:46.139052    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:46.239195 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:46.239168    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:46.306823 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:46.306792    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:47.472691 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:47.472655    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:47.566834 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:47.566799    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:47.874572 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:47.874454    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"]
Sep 16 19:50:47.877299 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:47.877270    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-bq8q4"]
Sep 16 19:50:48.734933 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:48.734895    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d652046-92b0-4364-8463-d3ea14240a92" path="/var/lib/kubelet/pods/9d652046-92b0-4364-8463-d3ea14240a92/volumes"
Sep 16 19:50:49.448487 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:49.448445    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:49.547251 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:49.547196    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:49.649078 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:49.649014    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:49.720416 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:49.720389    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:50.157855 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:50.157732    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:52.498133 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:52.498100    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:56.136985 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:56.136942    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:56.236999 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:50:56.236970    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:10.285675 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.285626    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:10.539997 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.539920    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"]
Sep 16 19:51:10.540382 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.540367    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79811896-54dd-4f30-9a37-f807f74dcd23" containerName="main"
Sep 16 19:51:10.540438 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.540384    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="79811896-54dd-4f30-9a37-f807f74dcd23" containerName="main"
Sep 16 19:51:10.540438 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.540401    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79811896-54dd-4f30-9a37-f807f74dcd23" containerName="storage-initializer"
Sep 16 19:51:10.540438 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.540406    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="79811896-54dd-4f30-9a37-f807f74dcd23" containerName="storage-initializer"
Sep 16 19:51:10.540537 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.540471    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="79811896-54dd-4f30-9a37-f807f74dcd23" containerName="main"
Sep 16 19:51:10.544917 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.544896    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"
Sep 16 19:51:10.547901 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.547878    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 19:51:10.548032 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.547878    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-mkkwr\""
Sep 16 19:51:10.549120 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.549077    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 19:51:10.549230 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.549156    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-jj75k\""
Sep 16 19:51:10.549230 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.549221    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 19:51:10.555969 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.555947    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"]
Sep 16 19:51:10.642804 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.642763    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:10.681346 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.681311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fddc211c-3ba8-4059-8aeb-800f30ae9f73-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d\" (UID: \"fddc211c-3ba8-4059-8aeb-800f30ae9f73\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"
Sep 16 19:51:10.681485 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.681354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhd69\" (UniqueName: \"kubernetes.io/projected/fddc211c-3ba8-4059-8aeb-800f30ae9f73-kube-api-access-rhd69\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d\" (UID: \"fddc211c-3ba8-4059-8aeb-800f30ae9f73\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"
Sep 16 19:51:10.783104 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.783039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fddc211c-3ba8-4059-8aeb-800f30ae9f73-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d\" (UID: \"fddc211c-3ba8-4059-8aeb-800f30ae9f73\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"
Sep 16 19:51:10.783264 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.783138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhd69\" (UniqueName: \"kubernetes.io/projected/fddc211c-3ba8-4059-8aeb-800f30ae9f73-kube-api-access-rhd69\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d\" (UID: \"fddc211c-3ba8-4059-8aeb-800f30ae9f73\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"
Sep 16 19:51:10.785833 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.785805    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fddc211c-3ba8-4059-8aeb-800f30ae9f73-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d\" (UID: \"fddc211c-3ba8-4059-8aeb-800f30ae9f73\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"
Sep 16 19:51:10.793089 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.792997    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhd69\" (UniqueName: \"kubernetes.io/projected/fddc211c-3ba8-4059-8aeb-800f30ae9f73-kube-api-access-rhd69\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d\" (UID: \"fddc211c-3ba8-4059-8aeb-800f30ae9f73\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"
Sep 16 19:51:10.855873 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:10.855844    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"
Sep 16 19:51:11.138133 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:11.138014    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"]
Sep 16 19:51:11.141014 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:51:11.140973    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddc211c_3ba8_4059_8aeb_800f30ae9f73.slice/crio-5cdd5f8759b810734157570a1c4a32d7bdd588b9a6c591eaec3d54bdacbf8768 WatchSource:0}: Error finding container 5cdd5f8759b810734157570a1c4a32d7bdd588b9a6c591eaec3d54bdacbf8768: Status 404 returned error can't find the container with id 5cdd5f8759b810734157570a1c4a32d7bdd588b9a6c591eaec3d54bdacbf8768
Sep 16 19:51:11.354743 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:11.354706    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d" event={"ID":"fddc211c-3ba8-4059-8aeb-800f30ae9f73","Type":"ContainerStarted","Data":"a0cd95cdff69124e49e38f4b2db341c80bedbe5ce3cc795eb81887626661ca83"}
Sep 16 19:51:11.355168 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:11.354764    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d" event={"ID":"fddc211c-3ba8-4059-8aeb-800f30ae9f73","Type":"ContainerStarted","Data":"5cdd5f8759b810734157570a1c4a32d7bdd588b9a6c591eaec3d54bdacbf8768"}
Sep 16 19:51:11.355168 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:11.354793    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"
Sep 16 19:51:11.384415 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:11.384361    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d" podStartSLOduration=1.384343898 podStartE2EDuration="1.384343898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:51:11.382043554 +0000 UTC m=+3971.184761772" watchObservedRunningTime="2026-09-16 19:51:11.384343898 +0000 UTC m=+3971.187062115"
Sep 16 19:51:13.555253 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:13.555174    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:15.757040 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:15.757008    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:22.093305 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:22.093269    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:23.051959 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:23.051918    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:35.075570 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:35.075536    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:37.380548 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:37.380510    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:37.494081 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:37.494020    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:42.162905 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.162869    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"]
Sep 16 19:51:42.167253 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.167233    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.170865 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.170841    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-hd8vj\""
Sep 16 19:51:42.170986 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.170888    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 16 19:51:42.170986 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.170917    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 16 19:51:42.171713 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.171698    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 16 19:51:42.177787 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.177759    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"]
Sep 16 19:51:42.194441 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.194408    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:42.265431 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.265402    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6624ad-4154-4bd0-af66-6a1245aba910-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.265577 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.265438    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.265577 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.265463    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.265577 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.265491    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.265577 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.265554    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.362089 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.362047    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"
Sep 16 19:51:42.366498 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.366469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.366498 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.366504    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.366771 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.366526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.366771 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.366607    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.366771 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.366639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzttf\" (UniqueName: \"kubernetes.io/projected/0c6624ad-4154-4bd0-af66-6a1245aba910-kube-api-access-mzttf\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.366771 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.366669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6624ad-4154-4bd0-af66-6a1245aba910-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.367104 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.366883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.367104 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.366970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.367104 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.367089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.369112 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.369088    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.369439 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.369422    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6624ad-4154-4bd0-af66-6a1245aba910-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.467204 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.467174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzttf\" (UniqueName: \"kubernetes.io/projected/0c6624ad-4154-4bd0-af66-6a1245aba910-kube-api-access-mzttf\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:42.481729 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:42.481703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzttf\" (UniqueName: \"kubernetes.io/projected/0c6624ad-4154-4bd0-af66-6a1245aba910-kube-api-access-mzttf\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:50.232114 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:50.232079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:50.247603 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:50.247575    2561 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:50.332585 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:50.332552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:50.332762 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:50.332695    2561 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:50.335773 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:50.335752    2561 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") DevicePath \"csi-7c4ce5c1eefd5d454d9c95c4f11f812ee7377dda48aa21311bec79dc0535da2a\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:50.425445 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:50.425403    2561 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/e3bf9cefc8fe9ee845aae35a5e89c33b4256cbb38db14601f2e77b7858f8a858/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:50.447543 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:50.447515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:50.579096 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:50.578983    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:51:50.862372 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:50.862342    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"]
Sep 16 19:51:50.863995 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:51:50.863968    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6624ad_4154_4bd0_af66_6a1245aba910.slice/crio-849f969e28d5dfc79025737665326875905036cfc7ece1aa4656ea67e399e70f WatchSource:0}: Error finding container 849f969e28d5dfc79025737665326875905036cfc7ece1aa4656ea67e399e70f: Status 404 returned error can't find the container with id 849f969e28d5dfc79025737665326875905036cfc7ece1aa4656ea67e399e70f
Sep 16 19:51:51.585179 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:51.585072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj" event={"ID":"0c6624ad-4154-4bd0-af66-6a1245aba910","Type":"ContainerStarted","Data":"ee85e80033b0429dbcca0d843152b51fa01e014f88c81fea98c03b86e195fcda"}
Sep 16 19:51:51.585179 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:51.585140    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj" event={"ID":"0c6624ad-4154-4bd0-af66-6a1245aba910","Type":"ContainerStarted","Data":"849f969e28d5dfc79025737665326875905036cfc7ece1aa4656ea67e399e70f"}
Sep 16 19:51:51.612157 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:51.612099    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj" podStartSLOduration=9.612083247 podStartE2EDuration="9.612083247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:51:51.60880953 +0000 UTC m=+4011.411527748" watchObservedRunningTime="2026-09-16 19:51:51.612083247 +0000 UTC m=+4011.414801457"
Sep 16 19:51:52.580070 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:51:52.580030    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:52:00.579770 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:52:00.579729    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:52:00.581431 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:52:00.581405    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj" podUID="0c6624ad-4154-4bd0-af66-6a1245aba910" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:52:10.579870 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:52:10.579822    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj" podUID="0c6624ad-4154-4bd0-af66-6a1245aba910" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:52:20.579706 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:52:20.579664    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj" podUID="0c6624ad-4154-4bd0-af66-6a1245aba910" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:52:26.385638 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:52:26.385547    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:26.450509 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:52:26.450471    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:30.580297 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:52:30.580248    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj" podUID="0c6624ad-4154-4bd0-af66-6a1245aba910" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:52:40.579480 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:52:40.579441    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj" podUID="0c6624ad-4154-4bd0-af66-6a1245aba910" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:52:50.580361 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:52:50.580311    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj" podUID="0c6624ad-4154-4bd0-af66-6a1245aba910" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:53:00.579518 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:00.579482    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj" podUID="0c6624ad-4154-4bd0-af66-6a1245aba910" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:53:10.590886 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:10.590846    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:53:10.598657 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:10.598636    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:53:20.139073 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:20.139026    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:20.161334 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:20.161300    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"]
Sep 16 19:53:21.275852 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:21.275810    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:22.362477 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:22.362427    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d" podUID="fddc211c-3ba8-4059-8aeb-800f30ae9f73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:53:24.066206 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:24.066169    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:32.362346 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:32.362305    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d" podUID="fddc211c-3ba8-4059-8aeb-800f30ae9f73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:53:35.184989 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:35.184904    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d" podUID="fddc211c-3ba8-4059-8aeb-800f30ae9f73" containerName="main" containerID="cri-o://a0cd95cdff69124e49e38f4b2db341c80bedbe5ce3cc795eb81887626661ca83" gracePeriod=45
Sep 16 19:53:38.850327 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:38.850280    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj_0c6624ad-4154-4bd0-af66-6a1245aba910/main/0.log"
Sep 16 19:53:40.165909 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:40.165867    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"]
Sep 16 19:53:40.182677 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:40.182630    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:42.363079 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:42.363029    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d" podUID="fddc211c-3ba8-4059-8aeb-800f30ae9f73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:53:42.363483 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:42.363153    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"
Sep 16 19:53:52.363129 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:52.363024    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d" podUID="fddc211c-3ba8-4059-8aeb-800f30ae9f73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:53:55.192612 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:55.192573    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj" podUID="0c6624ad-4154-4bd0-af66-6a1245aba910" containerName="main" containerID="cri-o://ee85e80033b0429dbcca0d843152b51fa01e014f88c81fea98c03b86e195fcda" gracePeriod=45
Sep 16 19:53:55.591698 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:55.591660    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj" podUID="0c6624ad-4154-4bd0-af66-6a1245aba910" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:53:55.844685 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:55.844623    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:53:56.005794 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.005767    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-home\") pod \"0c6624ad-4154-4bd0-af66-6a1245aba910\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") "
Sep 16 19:53:56.005984 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.005848    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6624ad-4154-4bd0-af66-6a1245aba910-tls-certs\") pod \"0c6624ad-4154-4bd0-af66-6a1245aba910\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") "
Sep 16 19:53:56.005984 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.005903    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-tmp-dir\") pod \"0c6624ad-4154-4bd0-af66-6a1245aba910\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") "
Sep 16 19:53:56.005984 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.005934    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzttf\" (UniqueName: \"kubernetes.io/projected/0c6624ad-4154-4bd0-af66-6a1245aba910-kube-api-access-mzttf\") pod \"0c6624ad-4154-4bd0-af66-6a1245aba910\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") "
Sep 16 19:53:56.005984 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.005969    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-model-cache\") pod \"0c6624ad-4154-4bd0-af66-6a1245aba910\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") "
Sep 16 19:53:56.006244 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.006003    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-dshm\") pod \"0c6624ad-4154-4bd0-af66-6a1245aba910\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") "
Sep 16 19:53:56.006244 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.006117    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"0c6624ad-4154-4bd0-af66-6a1245aba910\" (UID: \"0c6624ad-4154-4bd0-af66-6a1245aba910\") "
Sep 16 19:53:56.006342 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.006274    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c6624ad-4154-4bd0-af66-6a1245aba910" (UID: "0c6624ad-4154-4bd0-af66-6a1245aba910"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:56.006455 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.006413    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:53:56.006584 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.006520    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-home" (OuterVolumeSpecName: "home") pod "0c6624ad-4154-4bd0-af66-6a1245aba910" (UID: "0c6624ad-4154-4bd0-af66-6a1245aba910"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:56.008631 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.008597    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6624ad-4154-4bd0-af66-6a1245aba910-kube-api-access-mzttf" (OuterVolumeSpecName: "kube-api-access-mzttf") pod "0c6624ad-4154-4bd0-af66-6a1245aba910" (UID: "0c6624ad-4154-4bd0-af66-6a1245aba910"). InnerVolumeSpecName "kube-api-access-mzttf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:53:56.008740 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.008712    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6624ad-4154-4bd0-af66-6a1245aba910-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c6624ad-4154-4bd0-af66-6a1245aba910" (UID: "0c6624ad-4154-4bd0-af66-6a1245aba910"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:53:56.008813 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.008790    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-dshm" (OuterVolumeSpecName: "dshm") pod "0c6624ad-4154-4bd0-af66-6a1245aba910" (UID: "0c6624ad-4154-4bd0-af66-6a1245aba910"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:56.014722 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.014700    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99" (OuterVolumeSpecName: "kserve-pvc-source") pod "0c6624ad-4154-4bd0-af66-6a1245aba910" (UID: "0c6624ad-4154-4bd0-af66-6a1245aba910"). InnerVolumeSpecName "pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:53:56.017138 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.017115    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c6624ad-4154-4bd0-af66-6a1245aba910" (UID: "0c6624ad-4154-4bd0-af66-6a1245aba910"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:56.106970 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.106911    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:53:56.106970 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.106935    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzttf\" (UniqueName: \"kubernetes.io/projected/0c6624ad-4154-4bd0-af66-6a1245aba910-kube-api-access-mzttf\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:53:56.106970 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.106944    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:53:56.106970 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.106969    2561 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") on node \"ip-10-0-129-125.ec2.internal\" "
Sep 16 19:53:56.107190 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.106978    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6624ad-4154-4bd0-af66-6a1245aba910-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:53:56.107190 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.106987    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6624ad-4154-4bd0-af66-6a1245aba910-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:53:56.137564 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.137548    2561 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99") on node "ip-10-0-129-125.ec2.internal"
Sep 16 19:53:56.207969 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.207946    2561 reconciler_common.go:299] "Volume detached for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"csi-7c4ce5c1eefd5d454d9c95c4f11f812ee7377dda48aa21311bec79dc0535da2a\""
Sep 16 19:53:56.356003 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.355969    2561 generic.go:356] "Generic (PLEG): container finished" podID="0c6624ad-4154-4bd0-af66-6a1245aba910" containerID="ee85e80033b0429dbcca0d843152b51fa01e014f88c81fea98c03b86e195fcda" exitCode=0
Sep 16 19:53:56.356180 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.356024    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"
Sep 16 19:53:56.356180 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.356080    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj" event={"ID":"0c6624ad-4154-4bd0-af66-6a1245aba910","Type":"ContainerDied","Data":"ee85e80033b0429dbcca0d843152b51fa01e014f88c81fea98c03b86e195fcda"}
Sep 16 19:53:56.356180 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.356155    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj" event={"ID":"0c6624ad-4154-4bd0-af66-6a1245aba910","Type":"ContainerDied","Data":"849f969e28d5dfc79025737665326875905036cfc7ece1aa4656ea67e399e70f"}
Sep 16 19:53:56.356180 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.356178    2561 scope.go:117] "RemoveContainer" containerID="ee85e80033b0429dbcca0d843152b51fa01e014f88c81fea98c03b86e195fcda"
Sep 16 19:53:56.366014 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.365995    2561 scope.go:117] "RemoveContainer" containerID="ee85e80033b0429dbcca0d843152b51fa01e014f88c81fea98c03b86e195fcda"
Sep 16 19:53:56.366309 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:53:56.366286    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee85e80033b0429dbcca0d843152b51fa01e014f88c81fea98c03b86e195fcda\": container with ID starting with ee85e80033b0429dbcca0d843152b51fa01e014f88c81fea98c03b86e195fcda not found: ID does not exist" containerID="ee85e80033b0429dbcca0d843152b51fa01e014f88c81fea98c03b86e195fcda"
Sep 16 19:53:56.366398 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.366313    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee85e80033b0429dbcca0d843152b51fa01e014f88c81fea98c03b86e195fcda"} err="failed to get container status \"ee85e80033b0429dbcca0d843152b51fa01e014f88c81fea98c03b86e195fcda\": rpc error: code = NotFound desc = could not find container \"ee85e80033b0429dbcca0d843152b51fa01e014f88c81fea98c03b86e195fcda\": container with ID starting with ee85e80033b0429dbcca0d843152b51fa01e014f88c81fea98c03b86e195fcda not found: ID does not exist"
Sep 16 19:53:56.381115 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.381090    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"]
Sep 16 19:53:56.386158 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.386138    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-g2qkj"]
Sep 16 19:53:56.734947 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:53:56.734912    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c6624ad-4154-4bd0-af66-6a1245aba910" path="/var/lib/kubelet/pods/0c6624ad-4154-4bd0-af66-6a1245aba910/volumes"
Sep 16 19:54:00.238692 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:00.238662    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:00.333787 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:00.333752    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:02.362806 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:02.362763    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d" podUID="fddc211c-3ba8-4059-8aeb-800f30ae9f73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:54:05.410671 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:05.410635    2561 generic.go:356] "Generic (PLEG): container finished" podID="fddc211c-3ba8-4059-8aeb-800f30ae9f73" containerID="a0cd95cdff69124e49e38f4b2db341c80bedbe5ce3cc795eb81887626661ca83" exitCode=0
Sep 16 19:54:05.411196 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:05.410710    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d" event={"ID":"fddc211c-3ba8-4059-8aeb-800f30ae9f73","Type":"ContainerDied","Data":"a0cd95cdff69124e49e38f4b2db341c80bedbe5ce3cc795eb81887626661ca83"}
Sep 16 19:54:05.436490 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:05.436453    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"
Sep 16 19:54:05.484328 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:05.484299    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fddc211c-3ba8-4059-8aeb-800f30ae9f73-tls-certs\") pod \"fddc211c-3ba8-4059-8aeb-800f30ae9f73\" (UID: \"fddc211c-3ba8-4059-8aeb-800f30ae9f73\") "
Sep 16 19:54:05.484459 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:05.484383    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhd69\" (UniqueName: \"kubernetes.io/projected/fddc211c-3ba8-4059-8aeb-800f30ae9f73-kube-api-access-rhd69\") pod \"fddc211c-3ba8-4059-8aeb-800f30ae9f73\" (UID: \"fddc211c-3ba8-4059-8aeb-800f30ae9f73\") "
Sep 16 19:54:05.486677 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:05.486655    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fddc211c-3ba8-4059-8aeb-800f30ae9f73-kube-api-access-rhd69" (OuterVolumeSpecName: "kube-api-access-rhd69") pod "fddc211c-3ba8-4059-8aeb-800f30ae9f73" (UID: "fddc211c-3ba8-4059-8aeb-800f30ae9f73"). InnerVolumeSpecName "kube-api-access-rhd69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:54:05.486777 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:05.486681    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fddc211c-3ba8-4059-8aeb-800f30ae9f73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fddc211c-3ba8-4059-8aeb-800f30ae9f73" (UID: "fddc211c-3ba8-4059-8aeb-800f30ae9f73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:54:05.585251 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:05.585222    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhd69\" (UniqueName: \"kubernetes.io/projected/fddc211c-3ba8-4059-8aeb-800f30ae9f73-kube-api-access-rhd69\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:54:05.585251 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:05.585248    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fddc211c-3ba8-4059-8aeb-800f30ae9f73-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:54:05.615914 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:05.615830    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:05.716986 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:05.716959    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:06.416942 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:06.416909    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d" event={"ID":"fddc211c-3ba8-4059-8aeb-800f30ae9f73","Type":"ContainerDied","Data":"5cdd5f8759b810734157570a1c4a32d7bdd588b9a6c591eaec3d54bdacbf8768"}
Sep 16 19:54:06.416942 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:06.416935    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"
Sep 16 19:54:06.417518 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:06.416967    2561 scope.go:117] "RemoveContainer" containerID="a0cd95cdff69124e49e38f4b2db341c80bedbe5ce3cc795eb81887626661ca83"
Sep 16 19:54:06.440257 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:06.440222    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"]
Sep 16 19:54:06.442523 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:06.442498    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche9dv2d"]
Sep 16 19:54:06.734626 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:06.734592    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fddc211c-3ba8-4059-8aeb-800f30ae9f73" path="/var/lib/kubelet/pods/fddc211c-3ba8-4059-8aeb-800f30ae9f73/volumes"
Sep 16 19:54:06.822606 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:06.822571    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:06.946856 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:06.946822    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:07.272924 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:07.272894    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:10.017281 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:10.017248    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:10.118123 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:10.118089    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:11.929575 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:11.929544    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:12.453296 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:12.453259    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:14.240260 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:14.240219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:14.341188 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:14.341152    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:17.174955 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:17.174924    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:20.991767 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:20.991733    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:23.149809 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:23.149774    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:23.263229 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:23.263201    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:26.711990 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.711944    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"]
Sep 16 19:54:26.712452 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.712423    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fddc211c-3ba8-4059-8aeb-800f30ae9f73" containerName="main"
Sep 16 19:54:26.712452 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.712439    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fddc211c-3ba8-4059-8aeb-800f30ae9f73" containerName="main"
Sep 16 19:54:26.712543 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.712455    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6624ad-4154-4bd0-af66-6a1245aba910" containerName="main"
Sep 16 19:54:26.712543 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.712461    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6624ad-4154-4bd0-af66-6a1245aba910" containerName="main"
Sep 16 19:54:26.712543 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.712536    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fddc211c-3ba8-4059-8aeb-800f30ae9f73" containerName="main"
Sep 16 19:54:26.712543 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.712544    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c6624ad-4154-4bd0-af66-6a1245aba910" containerName="main"
Sep 16 19:54:26.715865 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.715840    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.718663 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.718638    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:54:26.718848 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.718741    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 19:54:26.719310 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.719286    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-lr9nf\""
Sep 16 19:54:26.719453 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.719432    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 19:54:26.719937 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.719914    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-xxrsk\""
Sep 16 19:54:26.720050 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.719920    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 19:54:26.720050 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.719975    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 19:54:26.738603 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.738576    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"]
Sep 16 19:54:26.764215 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.764188    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-dshm\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.764340 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.764225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7caee5b0-2f85-484d-832a-9290d0082182-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.764340 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.764257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7caee5b0-2f85-484d-832a-9290d0082182-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.764434 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.764332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scpc9\" (UniqueName: \"kubernetes.io/projected/7caee5b0-2f85-484d-832a-9290d0082182-kube-api-access-scpc9\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.764434 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.764376    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.764434 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.764413    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.764434 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.764429    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-model-cache\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.764610 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.764448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-home\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.802732 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.802698    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:26.865020 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.864990    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7caee5b0-2f85-484d-832a-9290d0082182-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.865170 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.865036    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpc9\" (UniqueName: \"kubernetes.io/projected/7caee5b0-2f85-484d-832a-9290d0082182-kube-api-access-scpc9\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.865170 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.865092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.865170 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.865126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.865170 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.865145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-model-cache\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.865170 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.865165    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-home\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.865408 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.865214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-dshm\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.865408 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.865246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7caee5b0-2f85-484d-832a-9290d0082182-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.865580 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.865553    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.865706 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.865652    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.865706 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.865667    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-model-cache\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.865840 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.865812    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-home\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.865949 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.865870    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7caee5b0-2f85-484d-832a-9290d0082182-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.867544 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.867523    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-dshm\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.867719 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.867702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7caee5b0-2f85-484d-832a-9290d0082182-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:26.875339 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:26.875311    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scpc9\" (UniqueName: \"kubernetes.io/projected/7caee5b0-2f85-484d-832a-9290d0082182-kube-api-access-scpc9\") pod \"custom-route-timeout-pd-test-kserve-545dd9468f-599j7\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:27.024224 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:27.024140    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:27.028045 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:27.028026    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:27.319745 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:27.319705    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"]
Sep 16 19:54:27.321483 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:54:27.321443    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7caee5b0_2f85_484d_832a_9290d0082182.slice/crio-a641d489673b22b4a4ffc560a3f624758755af5198bad8c64a7d85651690d018 WatchSource:0}: Error finding container a641d489673b22b4a4ffc560a3f624758755af5198bad8c64a7d85651690d018: Status 404 returned error can't find the container with id a641d489673b22b4a4ffc560a3f624758755af5198bad8c64a7d85651690d018
Sep 16 19:54:27.323760 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:27.323741    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:54:27.537980 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:27.537937    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" event={"ID":"7caee5b0-2f85-484d-832a-9290d0082182","Type":"ContainerStarted","Data":"a641d489673b22b4a4ffc560a3f624758755af5198bad8c64a7d85651690d018"}
Sep 16 19:54:29.551652 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:29.551610    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" event={"ID":"7caee5b0-2f85-484d-832a-9290d0082182","Type":"ContainerStarted","Data":"a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f"}
Sep 16 19:54:29.552107 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:29.551691    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:30.559297 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:30.559260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" event={"ID":"7caee5b0-2f85-484d-832a-9290d0082182","Type":"ContainerStarted","Data":"4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5"}
Sep 16 19:54:32.570769 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:32.570722    2561 generic.go:356] "Generic (PLEG): container finished" podID="7caee5b0-2f85-484d-832a-9290d0082182" containerID="4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5" exitCode=0
Sep 16 19:54:32.571203 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:32.570793    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" event={"ID":"7caee5b0-2f85-484d-832a-9290d0082182","Type":"ContainerDied","Data":"4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5"}
Sep 16 19:54:33.578588 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:33.578550    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" event={"ID":"7caee5b0-2f85-484d-832a-9290d0082182","Type":"ContainerStarted","Data":"d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701"}
Sep 16 19:54:33.610680 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:33.610614    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" podStartSLOduration=3.50193939 podStartE2EDuration="7.61059128s" totalImagesPullingTime="1.877077401s" totalInitContainerRuntime="2.231574489s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:54:33.609625851 +0000 UTC m=+4173.412344069" watchObservedRunningTime="2026-09-16 19:54:33.61059128 +0000 UTC m=+4173.413309498"
Sep 16 19:54:37.028839 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:37.028796    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:37.029364 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:37.028877    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:37.030361 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:37.030330    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:54:44.804422 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:44.804388    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:44.880298 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:44.880258    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:47.029303 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:47.029259    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:54:47.042371 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:47.042343    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:54:48.500106 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:48.500045    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:57.029396 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:54:57.029342    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:55:01.197573 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:55:01.197543    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:55:01.213452 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:55:01.213428    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 19:55:01.413559 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:55:01.413530    2561 scope.go:117] "RemoveContainer" containerID="53c95d3535af5b311a57c165480f399efb4474a8fce1721d49bca09e10f83dd5"
Sep 16 19:55:07.028692 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:55:07.028642    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:55:11.932110 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:55:11.932074    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:17.029117 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:55:17.029032    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:55:27.029308 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:55:27.029220    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:55:37.029394 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:55:37.029345    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:55:47.029312 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:55:47.029266    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:55:57.029542 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:55:57.029477    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:56:07.029021 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:07.028973    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:56:17.000795 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:17.000757    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:17.029144 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:17.029104    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:56:17.119196 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:17.119161    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:27.043787 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:27.043752    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:56:27.055813 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:27.055786    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:56:36.267200 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:36.267159    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:36.308874 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:36.308836    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"]
Sep 16 19:56:36.395869 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:36.395832    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:44.423857 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:44.423818    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:48.470377 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:48.470330    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:51.332808 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:51.332713    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main" containerID="cri-o://d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701" gracePeriod=45
Sep 16 19:56:51.850331 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:51.850275    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="llm-d-routing-sidecar" containerID="cri-o://a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f" gracePeriod=45
Sep 16 19:56:52.101304 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.101278    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:56:52.139706 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.139669    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7caee5b0-2f85-484d-832a-9290d0082182-cabundle-cert\") pod \"7caee5b0-2f85-484d-832a-9290d0082182\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") "
Sep 16 19:56:52.139896 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.139763    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7caee5b0-2f85-484d-832a-9290d0082182-tls-certs\") pod \"7caee5b0-2f85-484d-832a-9290d0082182\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") "
Sep 16 19:56:52.139896 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.139809    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scpc9\" (UniqueName: \"kubernetes.io/projected/7caee5b0-2f85-484d-832a-9290d0082182-kube-api-access-scpc9\") pod \"7caee5b0-2f85-484d-832a-9290d0082182\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") "
Sep 16 19:56:52.139896 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.139841    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-tmp-dir\") pod \"7caee5b0-2f85-484d-832a-9290d0082182\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") "
Sep 16 19:56:52.140107 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.139968    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-model-cache\") pod \"7caee5b0-2f85-484d-832a-9290d0082182\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") "
Sep 16 19:56:52.140107 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.140090    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-home\") pod \"7caee5b0-2f85-484d-832a-9290d0082182\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") "
Sep 16 19:56:52.140223 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.140124    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7caee5b0-2f85-484d-832a-9290d0082182-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7caee5b0-2f85-484d-832a-9290d0082182" (UID: "7caee5b0-2f85-484d-832a-9290d0082182"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:56:52.140223 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.140139    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-dshm\") pod \"7caee5b0-2f85-484d-832a-9290d0082182\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") "
Sep 16 19:56:52.140326 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.140225    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-kserve-provision-location\") pod \"7caee5b0-2f85-484d-832a-9290d0082182\" (UID: \"7caee5b0-2f85-484d-832a-9290d0082182\") "
Sep 16 19:56:52.140440 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.140419    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-model-cache" (OuterVolumeSpecName: "model-cache") pod "7caee5b0-2f85-484d-832a-9290d0082182" (UID: "7caee5b0-2f85-484d-832a-9290d0082182"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:52.140633 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.140614    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7caee5b0-2f85-484d-832a-9290d0082182-cabundle-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.140752 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.140638    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.140886 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.140861    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-home" (OuterVolumeSpecName: "home") pod "7caee5b0-2f85-484d-832a-9290d0082182" (UID: "7caee5b0-2f85-484d-832a-9290d0082182"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:52.142806 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.142779    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-dshm" (OuterVolumeSpecName: "dshm") pod "7caee5b0-2f85-484d-832a-9290d0082182" (UID: "7caee5b0-2f85-484d-832a-9290d0082182"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:52.143028 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.142995    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7caee5b0-2f85-484d-832a-9290d0082182-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7caee5b0-2f85-484d-832a-9290d0082182" (UID: "7caee5b0-2f85-484d-832a-9290d0082182"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:56:52.143028 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.143003    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7caee5b0-2f85-484d-832a-9290d0082182-kube-api-access-scpc9" (OuterVolumeSpecName: "kube-api-access-scpc9") pod "7caee5b0-2f85-484d-832a-9290d0082182" (UID: "7caee5b0-2f85-484d-832a-9290d0082182"). InnerVolumeSpecName "kube-api-access-scpc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:56:52.152859 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.152830    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7caee5b0-2f85-484d-832a-9290d0082182" (UID: "7caee5b0-2f85-484d-832a-9290d0082182"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:52.164228 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.164198    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7caee5b0-2f85-484d-832a-9290d0082182" (UID: "7caee5b0-2f85-484d-832a-9290d0082182"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:52.241239 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.241210    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7caee5b0-2f85-484d-832a-9290d0082182-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.241239 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.241238    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scpc9\" (UniqueName: \"kubernetes.io/projected/7caee5b0-2f85-484d-832a-9290d0082182-kube-api-access-scpc9\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.241396 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.241253    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.241396 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.241265    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.241396 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.241274    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.241396 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.241286    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7caee5b0-2f85-484d-832a-9290d0082182-kserve-provision-location\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.378333 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.378250    2561 generic.go:356] "Generic (PLEG): container finished" podID="7caee5b0-2f85-484d-832a-9290d0082182" containerID="d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701" exitCode=0
Sep 16 19:56:52.378333 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.378280    2561 generic.go:356] "Generic (PLEG): container finished" podID="7caee5b0-2f85-484d-832a-9290d0082182" containerID="a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f" exitCode=0
Sep 16 19:56:52.378333 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.378328    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"
Sep 16 19:56:52.378821 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.378326    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" event={"ID":"7caee5b0-2f85-484d-832a-9290d0082182","Type":"ContainerDied","Data":"d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701"}
Sep 16 19:56:52.378821 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.378411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" event={"ID":"7caee5b0-2f85-484d-832a-9290d0082182","Type":"ContainerDied","Data":"a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f"}
Sep 16 19:56:52.378821 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.378430    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7" event={"ID":"7caee5b0-2f85-484d-832a-9290d0082182","Type":"ContainerDied","Data":"a641d489673b22b4a4ffc560a3f624758755af5198bad8c64a7d85651690d018"}
Sep 16 19:56:52.378821 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.378443    2561 scope.go:117] "RemoveContainer" containerID="d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701"
Sep 16 19:56:52.388269 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.388193    2561 scope.go:117] "RemoveContainer" containerID="4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5"
Sep 16 19:56:52.398646 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.398625    2561 scope.go:117] "RemoveContainer" containerID="a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f"
Sep 16 19:56:52.404359 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.404338    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"]
Sep 16 19:56:52.408070 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.408036    2561 scope.go:117] "RemoveContainer" containerID="d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701"
Sep 16 19:56:52.408367 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:56:52.408349    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701\": container with ID starting with d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701 not found: ID does not exist" containerID="d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701"
Sep 16 19:56:52.408431 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.408375    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701"} err="failed to get container status \"d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701\": rpc error: code = NotFound desc = could not find container \"d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701\": container with ID starting with d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701 not found: ID does not exist"
Sep 16 19:56:52.408431 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.408392    2561 scope.go:117] "RemoveContainer" containerID="4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5"
Sep 16 19:56:52.408672 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:56:52.408650    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5\": container with ID starting with 4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5 not found: ID does not exist" containerID="4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5"
Sep 16 19:56:52.408731 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.408676    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5"} err="failed to get container status \"4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5\": rpc error: code = NotFound desc = could not find container \"4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5\": container with ID starting with 4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5 not found: ID does not exist"
Sep 16 19:56:52.408731 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.408688    2561 scope.go:117] "RemoveContainer" containerID="a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f"
Sep 16 19:56:52.408943 ip-10-0-129-125 kubenswrapper[2561]: E0916 19:56:52.408923    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f\": container with ID starting with a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f not found: ID does not exist" containerID="a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f"
Sep 16 19:56:52.408987 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.408949    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f"} err="failed to get container status \"a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f\": rpc error: code = NotFound desc = could not find container \"a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f\": container with ID starting with a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f not found: ID does not exist"
Sep 16 19:56:52.408987 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.408965    2561 scope.go:117] "RemoveContainer" containerID="d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701"
Sep 16 19:56:52.409267 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.409249    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701"} err="failed to get container status \"d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701\": rpc error: code = NotFound desc = could not find container \"d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701\": container with ID starting with d53cc03218a7a4027e805b8e5e291a628ed6357941e8b2e2b95b56a28122a701 not found: ID does not exist"
Sep 16 19:56:52.409334 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.409268    2561 scope.go:117] "RemoveContainer" containerID="4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5"
Sep 16 19:56:52.409541 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.409524    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5"} err="failed to get container status \"4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5\": rpc error: code = NotFound desc = could not find container \"4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5\": container with ID starting with 4308fcb752441640ce640cbbba2b68d569116acad57f93be5b718b7cb669ddb5 not found: ID does not exist"
Sep 16 19:56:52.409593 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.409541    2561 scope.go:117] "RemoveContainer" containerID="a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f"
Sep 16 19:56:52.409662 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.409642    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-545dd9468f-599j7"]
Sep 16 19:56:52.409791 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.409773    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f"} err="failed to get container status \"a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f\": rpc error: code = NotFound desc = could not find container \"a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f\": container with ID starting with a5b121dba1232d99bae516fa3e6be9980186443368d1c7fa47fff5766d7d212f not found: ID does not exist"
Sep 16 19:56:52.733993 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:56:52.733961    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7caee5b0-2f85-484d-832a-9290d0082182" path="/var/lib/kubelet/pods/7caee5b0-2f85-484d-832a-9290d0082182/volumes"
Sep 16 19:57:07.174928 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:07.174887    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:07.424545 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:07.424511    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:13.844243 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:13.844208    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:20.253030 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:20.252984    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:20.352904 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:20.352873    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:25.660356 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:25.660320    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:25.930981 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:25.930898    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:26.379136 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.379088    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:26.557430 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.557388    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:26.842932 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.841593    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"]
Sep 16 19:57:26.842932 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.842204    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="llm-d-routing-sidecar"
Sep 16 19:57:26.842932 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.842224    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="llm-d-routing-sidecar"
Sep 16 19:57:26.842932 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.842263    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main"
Sep 16 19:57:26.842932 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.842273    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main"
Sep 16 19:57:26.842932 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.842293    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="storage-initializer"
Sep 16 19:57:26.842932 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.842303    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="storage-initializer"
Sep 16 19:57:26.842932 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.842399    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="llm-d-routing-sidecar"
Sep 16 19:57:26.842932 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.842411    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7caee5b0-2f85-484d-832a-9290d0082182" containerName="main"
Sep 16 19:57:26.846640 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.846617    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:26.852906 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.852881    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-dnbk7\""
Sep 16 19:57:26.853355 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.853331    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 19:57:26.854824 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.854807    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 19:57:26.864720 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.864699    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:57:26.904658 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.904633    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"]
Sep 16 19:57:26.944168 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.944142    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-workload-socket\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:26.944277 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.944175    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-workload-certs\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:26.944277 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.944197    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-credential-socket\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:26.944277 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.944247    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db46328-4d57-410a-891c-f89a73e1c941-istio-token\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:26.944402 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.944302    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db46328-4d57-410a-891c-f89a73e1c941-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:26.944402 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.944331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbsmc\" (UniqueName: \"kubernetes.io/projected/2db46328-4d57-410a-891c-f89a73e1c941-kube-api-access-jbsmc\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:26.944402 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.944354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-istio-envoy\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:26.944402 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.944390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-istio-data\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:26.944534 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.944417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db46328-4d57-410a-891c-f89a73e1c941-istio-podinfo\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:26.973818 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:26.973781    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:27.045669 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.045630    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db46328-4d57-410a-891c-f89a73e1c941-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.045669 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.045674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbsmc\" (UniqueName: \"kubernetes.io/projected/2db46328-4d57-410a-891c-f89a73e1c941-kube-api-access-jbsmc\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.045911 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.045698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-istio-envoy\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.045911 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.045858    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-istio-data\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.046027 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.045917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db46328-4d57-410a-891c-f89a73e1c941-istio-podinfo\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.046027 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.045999    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-workload-socket\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.046173 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.046031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-workload-certs\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.046173 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.046085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-credential-socket\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.046173 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.046118    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db46328-4d57-410a-891c-f89a73e1c941-istio-token\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.046532 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.046505    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db46328-4d57-410a-891c-f89a73e1c941-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.046720 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.046690    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-istio-data\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.046940 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.046888    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-credential-socket\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.046940 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.046924    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-workload-socket\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.047137 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.047013    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-workload-certs\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.048659 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.048633    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-istio-envoy\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.049340 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.049319    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db46328-4d57-410a-891c-f89a73e1c941-istio-podinfo\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.067626 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.067605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbsmc\" (UniqueName: \"kubernetes.io/projected/2db46328-4d57-410a-891c-f89a73e1c941-kube-api-access-jbsmc\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.067779 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.067694    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db46328-4d57-410a-891c-f89a73e1c941-istio-token\") pod \"router-gateway-2-openshift-default-748c4bc748-8xdw5\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.161008 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.160926    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:27.241557 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.241521    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:27.448100 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.448075    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"]
Sep 16 19:57:27.450531 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:57:27.450500    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db46328_4d57_410a_891c_f89a73e1c941.slice/crio-3c186824d5f0864cb3752610124d7a08af04e666e7126eab9b7dfe6364764624 WatchSource:0}: Error finding container 3c186824d5f0864cb3752610124d7a08af04e666e7126eab9b7dfe6364764624: Status 404 returned error can't find the container with id 3c186824d5f0864cb3752610124d7a08af04e666e7126eab9b7dfe6364764624
Sep 16 19:57:27.569797 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:27.569763    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5" event={"ID":"2db46328-4d57-410a-891c-f89a73e1c941","Type":"ContainerStarted","Data":"3c186824d5f0864cb3752610124d7a08af04e666e7126eab9b7dfe6364764624"}
Sep 16 19:57:30.196796 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:30.196762    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:30.411773 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:30.411723    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:57:30.411916 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:30.411811    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:57:30.411916 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:30.411839    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:57:30.592711 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:30.592662    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5" event={"ID":"2db46328-4d57-410a-891c-f89a73e1c941","Type":"ContainerStarted","Data":"d53b52eacec25d614e3559eb7a6e5775e762a8aee2911fc27018edb787495661"}
Sep 16 19:57:30.618754 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:30.618684    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5" podStartSLOduration=1.6597185140000001 podStartE2EDuration="4.618662157s" totalImagesPullingTime="2.958943643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:30.616262185 +0000 UTC m=+4350.418980403" watchObservedRunningTime="2026-09-16 19:57:30.618662157 +0000 UTC m=+4350.421380376"
Sep 16 19:57:31.161547 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:31.161508    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:31.163323 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:31.163288    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5" podUID="2db46328-4d57-410a-891c-f89a73e1c941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 16 19:57:32.162193 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:32.162152    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5" podUID="2db46328-4d57-410a-891c-f89a73e1c941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 16 19:57:33.162379 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:33.162333    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5" podUID="2db46328-4d57-410a-891c-f89a73e1c941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 16 19:57:33.600758 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:33.600716    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:34.161699 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:34.161656    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5" podUID="2db46328-4d57-410a-891c-f89a73e1c941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 16 19:57:35.162140 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:35.162092    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5" podUID="2db46328-4d57-410a-891c-f89a73e1c941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 16 19:57:36.162346 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:36.162300    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5" podUID="2db46328-4d57-410a-891c-f89a73e1c941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 16 19:57:37.165295 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:37.165264    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:37.165759 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:37.165543    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:37.166356 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:37.166338    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 19:57:39.640482 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:39.640438    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:39.718231 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:39.718198    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:55.605628 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:57:55.605576    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:12.953298 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:12.953263    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"]
Sep 16 19:58:12.961991 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:12.961965    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:12.965427 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:12.965398    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 19:58:12.965615 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:12.965426    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-ttkd6\""
Sep 16 19:58:12.965738 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:12.965707    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:58:12.967258 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:12.967236    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 19:58:12.967379 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:12.967348    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-p6ddw\""
Sep 16 19:58:12.982848 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:12.982817    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"]
Sep 16 19:58:13.028776 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.028743    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:13.069395 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.069359    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-model-cache\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.069395 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.069400    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-dshm\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.069654 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.069425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.069654 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.069512    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hw4n\" (UniqueName: \"kubernetes.io/projected/9af82cb1-3e57-4cbd-b39e-b40286f611bd-kube-api-access-2hw4n\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.069654 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.069559    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-home\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.069654 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.069609    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-tmp-dir\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.069654 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.069645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9af82cb1-3e57-4cbd-b39e-b40286f611bd-tls-certs\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.069877 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.069674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9af82cb1-3e57-4cbd-b39e-b40286f611bd-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.170957 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.170919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-model-cache\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.171183 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.170970    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-dshm\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.171183 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.171002    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.171183 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.171029    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hw4n\" (UniqueName: \"kubernetes.io/projected/9af82cb1-3e57-4cbd-b39e-b40286f611bd-kube-api-access-2hw4n\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.171183 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.171083    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-home\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.171183 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.171124    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-tmp-dir\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.171183 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.171151    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9af82cb1-3e57-4cbd-b39e-b40286f611bd-tls-certs\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.171183 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.171176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9af82cb1-3e57-4cbd-b39e-b40286f611bd-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.171650 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.171458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-model-cache\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.171650 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.171508    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.171650 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.171625    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-tmp-dir\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.171836 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.171669    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-home\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.171949 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.171929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9af82cb1-3e57-4cbd-b39e-b40286f611bd-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.173523 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.173497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-dshm\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.173893 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.173872    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9af82cb1-3e57-4cbd-b39e-b40286f611bd-tls-certs\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.180413 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.180386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hw4n\" (UniqueName: \"kubernetes.io/projected/9af82cb1-3e57-4cbd-b39e-b40286f611bd-kube-api-access-2hw4n\") pod \"router-with-refs-pd-test-kserve-85c7697cf-l27vm\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.274920 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.274887    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:13.562587 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.562558    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"]
Sep 16 19:58:13.565605 ip-10-0-129-125 kubenswrapper[2561]: W0916 19:58:13.565581    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af82cb1_3e57_4cbd_b39e_b40286f611bd.slice/crio-acbd99eb4555442a276c285d616fa9bc2eb3009f3440c137050a8800ebea9471 WatchSource:0}: Error finding container acbd99eb4555442a276c285d616fa9bc2eb3009f3440c137050a8800ebea9471: Status 404 returned error can't find the container with id acbd99eb4555442a276c285d616fa9bc2eb3009f3440c137050a8800ebea9471
Sep 16 19:58:13.648824 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.648785    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:13.756745 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.756709    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:13.842580 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.842466    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" event={"ID":"9af82cb1-3e57-4cbd-b39e-b40286f611bd","Type":"ContainerStarted","Data":"4610f0605155f5b2cec3beef88e24e3d1cf45f0328aea4be32ea54a85f2fcdd1"}
Sep 16 19:58:13.842580 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.842542    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" event={"ID":"9af82cb1-3e57-4cbd-b39e-b40286f611bd","Type":"ContainerStarted","Data":"acbd99eb4555442a276c285d616fa9bc2eb3009f3440c137050a8800ebea9471"}
Sep 16 19:58:13.842580 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:13.842577    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:14.849724 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:14.849685    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" event={"ID":"9af82cb1-3e57-4cbd-b39e-b40286f611bd","Type":"ContainerStarted","Data":"4eec6f541c44ec8b7864a4d12bfe87cc3a58c3d10aa01f0bd83aad5808e5e4a1"}
Sep 16 19:58:16.863107 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:16.863047    2561 generic.go:356] "Generic (PLEG): container finished" podID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerID="4eec6f541c44ec8b7864a4d12bfe87cc3a58c3d10aa01f0bd83aad5808e5e4a1" exitCode=0
Sep 16 19:58:16.863530 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:16.863114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" event={"ID":"9af82cb1-3e57-4cbd-b39e-b40286f611bd","Type":"ContainerDied","Data":"4eec6f541c44ec8b7864a4d12bfe87cc3a58c3d10aa01f0bd83aad5808e5e4a1"}
Sep 16 19:58:17.871566 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:17.871531    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" event={"ID":"9af82cb1-3e57-4cbd-b39e-b40286f611bd","Type":"ContainerStarted","Data":"4b5b9ef8f9586329dfce1e45cfc5557562d7cde5f909fd91a92ee087a97fc221"}
Sep 16 19:58:17.897826 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:17.897763    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" podStartSLOduration=3.472649996 podStartE2EDuration="5.897747281s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.425097285s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:58:17.895909745 +0000 UTC m=+4397.698627964" watchObservedRunningTime="2026-09-16 19:58:17.897747281 +0000 UTC m=+4397.700465498"
Sep 16 19:58:23.275142 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:23.275033    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:23.275142 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:23.275132    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:23.276676 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:23.276638    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:58:23.288699 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:23.288679    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 19:58:33.275265 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:33.275221    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:58:43.276202 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:43.276151    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:58:53.276345 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:58:53.276287    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:59:03.275754 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:59:03.275700    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:59:13.275552 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:59:13.275490    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:59:23.276108 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:59:23.276021    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:59:33.276352 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:59:33.276298    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:59:43.275789 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:59:43.275730    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:59:50.351240 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:59:50.351195    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:59:51.909816 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:59:51.909735    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 19:59:53.275853 ip-10-0-129-125 kubenswrapper[2561]: I0916 19:59:53.275804    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 20:00:01.239288 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:01.239261    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 20:00:01.255425 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:01.255401    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 20:00:03.275988 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:03.275939    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 20:00:13.254886 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:13.254848    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:13.284889 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:13.284863    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 20:00:13.297394 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:13.297373    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 20:00:13.346522 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:13.346485    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:22.848743 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:22.848710    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:22.873977 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:22.873945    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"]
Sep 16 20:00:22.954366 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:22.947751    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:37.899246 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:37.899184    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main" containerID="cri-o://4b5b9ef8f9586329dfce1e45cfc5557562d7cde5f909fd91a92ee087a97fc221" gracePeriod=45
Sep 16 20:00:38.511348 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.511307    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="llm-d-routing-sidecar" containerID="cri-o://4610f0605155f5b2cec3beef88e24e3d1cf45f0328aea4be32ea54a85f2fcdd1" gracePeriod=45
Sep 16 20:00:38.696685 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.696591    2561 generic.go:356] "Generic (PLEG): container finished" podID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerID="4b5b9ef8f9586329dfce1e45cfc5557562d7cde5f909fd91a92ee087a97fc221" exitCode=0
Sep 16 20:00:38.696685 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.696622    2561 generic.go:356] "Generic (PLEG): container finished" podID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerID="4610f0605155f5b2cec3beef88e24e3d1cf45f0328aea4be32ea54a85f2fcdd1" exitCode=0
Sep 16 20:00:38.696685 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.696624    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" event={"ID":"9af82cb1-3e57-4cbd-b39e-b40286f611bd","Type":"ContainerDied","Data":"4b5b9ef8f9586329dfce1e45cfc5557562d7cde5f909fd91a92ee087a97fc221"}
Sep 16 20:00:38.696901 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.696701    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" event={"ID":"9af82cb1-3e57-4cbd-b39e-b40286f611bd","Type":"ContainerDied","Data":"4610f0605155f5b2cec3beef88e24e3d1cf45f0328aea4be32ea54a85f2fcdd1"}
Sep 16 20:00:38.760420 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.760391    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 20:00:38.857009 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.856973    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9af82cb1-3e57-4cbd-b39e-b40286f611bd-tls-certs\") pod \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") "
Sep 16 20:00:38.857221 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.857045    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-dshm\") pod \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") "
Sep 16 20:00:38.857221 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.857093    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9af82cb1-3e57-4cbd-b39e-b40286f611bd-cabundle-cert\") pod \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") "
Sep 16 20:00:38.857221 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.857124    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-home\") pod \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") "
Sep 16 20:00:38.857221 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.857172    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-model-cache\") pod \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") "
Sep 16 20:00:38.857221 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.857212    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-kserve-provision-location\") pod \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") "
Sep 16 20:00:38.857485 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.857240    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hw4n\" (UniqueName: \"kubernetes.io/projected/9af82cb1-3e57-4cbd-b39e-b40286f611bd-kube-api-access-2hw4n\") pod \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") "
Sep 16 20:00:38.857485 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.857266    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-tmp-dir\") pod \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\" (UID: \"9af82cb1-3e57-4cbd-b39e-b40286f611bd\") "
Sep 16 20:00:38.857591 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.857479    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "9af82cb1-3e57-4cbd-b39e-b40286f611bd" (UID: "9af82cb1-3e57-4cbd-b39e-b40286f611bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:38.857591 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.857529    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af82cb1-3e57-4cbd-b39e-b40286f611bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9af82cb1-3e57-4cbd-b39e-b40286f611bd" (UID: "9af82cb1-3e57-4cbd-b39e-b40286f611bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:00:38.857732 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.857714    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9af82cb1-3e57-4cbd-b39e-b40286f611bd-cabundle-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:00:38.857826 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.857779    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:00:38.858025 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.858000    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-home" (OuterVolumeSpecName: "home") pod "9af82cb1-3e57-4cbd-b39e-b40286f611bd" (UID: "9af82cb1-3e57-4cbd-b39e-b40286f611bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:38.859572 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.859546    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-dshm" (OuterVolumeSpecName: "dshm") pod "9af82cb1-3e57-4cbd-b39e-b40286f611bd" (UID: "9af82cb1-3e57-4cbd-b39e-b40286f611bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:38.859687 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.859615    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af82cb1-3e57-4cbd-b39e-b40286f611bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9af82cb1-3e57-4cbd-b39e-b40286f611bd" (UID: "9af82cb1-3e57-4cbd-b39e-b40286f611bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:00:38.859833 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.859816    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af82cb1-3e57-4cbd-b39e-b40286f611bd-kube-api-access-2hw4n" (OuterVolumeSpecName: "kube-api-access-2hw4n") pod "9af82cb1-3e57-4cbd-b39e-b40286f611bd" (UID: "9af82cb1-3e57-4cbd-b39e-b40286f611bd"). InnerVolumeSpecName "kube-api-access-2hw4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:00:38.869793 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.869769    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9af82cb1-3e57-4cbd-b39e-b40286f611bd" (UID: "9af82cb1-3e57-4cbd-b39e-b40286f611bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:38.880485 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.880461    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9af82cb1-3e57-4cbd-b39e-b40286f611bd" (UID: "9af82cb1-3e57-4cbd-b39e-b40286f611bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:38.958926 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.958844    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-kserve-provision-location\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:00:38.958926 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.958872    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hw4n\" (UniqueName: \"kubernetes.io/projected/9af82cb1-3e57-4cbd-b39e-b40286f611bd-kube-api-access-2hw4n\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:00:38.958926 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.958886    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:00:38.958926 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.958898    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9af82cb1-3e57-4cbd-b39e-b40286f611bd-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:00:38.958926 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.958908    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:00:38.958926 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:38.958918    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9af82cb1-3e57-4cbd-b39e-b40286f611bd-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:00:39.704308 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:39.704279    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"
Sep 16 20:00:39.704308 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:39.704295    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm" event={"ID":"9af82cb1-3e57-4cbd-b39e-b40286f611bd","Type":"ContainerDied","Data":"acbd99eb4555442a276c285d616fa9bc2eb3009f3440c137050a8800ebea9471"}
Sep 16 20:00:39.704566 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:39.704353    2561 scope.go:117] "RemoveContainer" containerID="4b5b9ef8f9586329dfce1e45cfc5557562d7cde5f909fd91a92ee087a97fc221"
Sep 16 20:00:39.716989 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:39.716925    2561 scope.go:117] "RemoveContainer" containerID="4eec6f541c44ec8b7864a4d12bfe87cc3a58c3d10aa01f0bd83aad5808e5e4a1"
Sep 16 20:00:39.726643 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:39.726619    2561 scope.go:117] "RemoveContainer" containerID="4610f0605155f5b2cec3beef88e24e3d1cf45f0328aea4be32ea54a85f2fcdd1"
Sep 16 20:00:39.738493 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:39.738469    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"]
Sep 16 20:00:39.743571 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:39.743548    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85c7697cf-l27vm"]
Sep 16 20:00:40.735211 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:40.735178    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" path="/var/lib/kubelet/pods/9af82cb1-3e57-4cbd-b39e-b40286f611bd/volumes"
Sep 16 20:00:51.720411 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:51.720371    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:51.831084 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:51.831031    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:58.150764 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:58.150726    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:58.407484 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:58.407397    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:58.472301 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:00:58.472267    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:04.126750 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:04.126714    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:04.230597 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:04.230561    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:05.416253 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.416215    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:05.628412 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.628379    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"]
Sep 16 20:01:05.628894 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.628870    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="storage-initializer"
Sep 16 20:01:05.628894 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.628894    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="storage-initializer"
Sep 16 20:01:05.629126 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.628914    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="llm-d-routing-sidecar"
Sep 16 20:01:05.629126 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.628923    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="llm-d-routing-sidecar"
Sep 16 20:01:05.629126 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.628970    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main"
Sep 16 20:01:05.629126 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.628979    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main"
Sep 16 20:01:05.629126 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.629119    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="llm-d-routing-sidecar"
Sep 16 20:01:05.629329 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.629132    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9af82cb1-3e57-4cbd-b39e-b40286f611bd" containerName="main"
Sep 16 20:01:05.633331 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.633315    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"
Sep 16 20:01:05.638335 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.638313    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 20:01:05.638461 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.638330    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 20:01:05.638461 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.638348    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 20:01:05.638461 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.638334    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-pxvng\""
Sep 16 20:01:05.638461 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.638381    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-pcmbs\""
Sep 16 20:01:05.646531 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.646509    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"]
Sep 16 20:01:05.667644 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.667586    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:05.708815 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.708787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ngxq\" (UniqueName: \"kubernetes.io/projected/caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898-kube-api-access-5ngxq\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh\" (UID: \"caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"
Sep 16 20:01:05.708918 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.708824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh\" (UID: \"caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"
Sep 16 20:01:05.809664 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.809627    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ngxq\" (UniqueName: \"kubernetes.io/projected/caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898-kube-api-access-5ngxq\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh\" (UID: \"caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"
Sep 16 20:01:05.809848 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.809670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh\" (UID: \"caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"
Sep 16 20:01:05.812618 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.812586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh\" (UID: \"caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"
Sep 16 20:01:05.819466 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.819442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ngxq\" (UniqueName: \"kubernetes.io/projected/caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898-kube-api-access-5ngxq\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh\" (UID: \"caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"
Sep 16 20:01:05.944271 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:05.944184    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"
Sep 16 20:01:06.230296 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:06.230264    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"]
Sep 16 20:01:06.232272 ip-10-0-129-125 kubenswrapper[2561]: W0916 20:01:06.232239    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf3ea1b_d1a9_4a9e_a4d7_1a8c11c84898.slice/crio-252e7d1b6f0e33ed9df5345532225a0b70bdd8ca946f0380d78148cb5ba981b7 WatchSource:0}: Error finding container 252e7d1b6f0e33ed9df5345532225a0b70bdd8ca946f0380d78148cb5ba981b7: Status 404 returned error can't find the container with id 252e7d1b6f0e33ed9df5345532225a0b70bdd8ca946f0380d78148cb5ba981b7
Sep 16 20:01:06.859595 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:06.859558    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh" event={"ID":"caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898","Type":"ContainerStarted","Data":"5bb5425ba04186e9a16a85493d9c10ea08e64d4dd763936ab7759bb0bc199139"}
Sep 16 20:01:06.860035 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:06.859620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh" event={"ID":"caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898","Type":"ContainerStarted","Data":"252e7d1b6f0e33ed9df5345532225a0b70bdd8ca946f0380d78148cb5ba981b7"}
Sep 16 20:01:06.860035 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:06.859649    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"
Sep 16 20:01:06.881026 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:06.880977    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh" podStartSLOduration=1.880963196 podStartE2EDuration="1.880963196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:01:06.87773728 +0000 UTC m=+4566.680455510" watchObservedRunningTime="2026-09-16 20:01:06.880963196 +0000 UTC m=+4566.683681412"
Sep 16 20:01:13.073560 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:13.073519    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:13.170199 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:13.170167    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:16.549194 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:16.549158    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:18.024870 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:18.024839    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"]
Sep 16 20:01:18.025342 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:18.025021    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5" podUID="2db46328-4d57-410a-891c-f89a73e1c941" containerName="istio-proxy" containerID="cri-o://d53b52eacec25d614e3559eb7a6e5775e762a8aee2911fc27018edb787495661" gracePeriod=30
Sep 16 20:01:18.082735 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:18.082692    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:22.168563 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:22.168477    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5" podUID="2db46328-4d57-410a-891c-f89a73e1c941" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 20:01:23.278921 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.278896    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 20:01:23.370729 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.370702    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db46328-4d57-410a-891c-f89a73e1c941-istiod-ca-cert\") pod \"2db46328-4d57-410a-891c-f89a73e1c941\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") "
Sep 16 20:01:23.370899 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.370764    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-istio-data\") pod \"2db46328-4d57-410a-891c-f89a73e1c941\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") "
Sep 16 20:01:23.370899 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.370850    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db46328-4d57-410a-891c-f89a73e1c941-istio-token\") pod \"2db46328-4d57-410a-891c-f89a73e1c941\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") "
Sep 16 20:01:23.370899 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.370881    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-workload-socket\") pod \"2db46328-4d57-410a-891c-f89a73e1c941\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") "
Sep 16 20:01:23.371109 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.370956    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db46328-4d57-410a-891c-f89a73e1c941-istio-podinfo\") pod \"2db46328-4d57-410a-891c-f89a73e1c941\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") "
Sep 16 20:01:23.371109 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.370992    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbsmc\" (UniqueName: \"kubernetes.io/projected/2db46328-4d57-410a-891c-f89a73e1c941-kube-api-access-jbsmc\") pod \"2db46328-4d57-410a-891c-f89a73e1c941\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") "
Sep 16 20:01:23.371109 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.371033    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-workload-certs\") pod \"2db46328-4d57-410a-891c-f89a73e1c941\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") "
Sep 16 20:01:23.371109 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.371075    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db46328-4d57-410a-891c-f89a73e1c941-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2db46328-4d57-410a-891c-f89a73e1c941" (UID: "2db46328-4d57-410a-891c-f89a73e1c941"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:01:23.371109 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.371087    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-istio-envoy\") pod \"2db46328-4d57-410a-891c-f89a73e1c941\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") "
Sep 16 20:01:23.371832 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.371183    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-credential-socket\") pod \"2db46328-4d57-410a-891c-f89a73e1c941\" (UID: \"2db46328-4d57-410a-891c-f89a73e1c941\") "
Sep 16 20:01:23.371832 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.371507    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db46328-4d57-410a-891c-f89a73e1c941-istiod-ca-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:23.371971 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.371944    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-istio-data" (OuterVolumeSpecName: "istio-data") pod "2db46328-4d57-410a-891c-f89a73e1c941" (UID: "2db46328-4d57-410a-891c-f89a73e1c941"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:23.372216 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.371979    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2db46328-4d57-410a-891c-f89a73e1c941" (UID: "2db46328-4d57-410a-891c-f89a73e1c941"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:23.372216 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.372050    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2db46328-4d57-410a-891c-f89a73e1c941" (UID: "2db46328-4d57-410a-891c-f89a73e1c941"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:23.372216 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.372148    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2db46328-4d57-410a-891c-f89a73e1c941" (UID: "2db46328-4d57-410a-891c-f89a73e1c941"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:23.373669 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.373644    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2db46328-4d57-410a-891c-f89a73e1c941" (UID: "2db46328-4d57-410a-891c-f89a73e1c941"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:23.373669 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.373663    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db46328-4d57-410a-891c-f89a73e1c941-istio-token" (OuterVolumeSpecName: "istio-token") pod "2db46328-4d57-410a-891c-f89a73e1c941" (UID: "2db46328-4d57-410a-891c-f89a73e1c941"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:23.374035 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.374003    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db46328-4d57-410a-891c-f89a73e1c941-kube-api-access-jbsmc" (OuterVolumeSpecName: "kube-api-access-jbsmc") pod "2db46328-4d57-410a-891c-f89a73e1c941" (UID: "2db46328-4d57-410a-891c-f89a73e1c941"). InnerVolumeSpecName "kube-api-access-jbsmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:23.374421 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.374401    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2db46328-4d57-410a-891c-f89a73e1c941-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2db46328-4d57-410a-891c-f89a73e1c941" (UID: "2db46328-4d57-410a-891c-f89a73e1c941"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 20:01:23.472938 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.472913    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-credential-socket\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:23.472938 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.472937    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-istio-data\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:23.473106 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.472947    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db46328-4d57-410a-891c-f89a73e1c941-istio-token\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:23.473106 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.472957    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-workload-socket\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:23.473106 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.472965    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db46328-4d57-410a-891c-f89a73e1c941-istio-podinfo\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:23.473106 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.472973    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbsmc\" (UniqueName: \"kubernetes.io/projected/2db46328-4d57-410a-891c-f89a73e1c941-kube-api-access-jbsmc\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:23.473106 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.472981    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-workload-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:23.473106 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.472989    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db46328-4d57-410a-891c-f89a73e1c941-istio-envoy\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:23.957691 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.957649    2561 generic.go:356] "Generic (PLEG): container finished" podID="2db46328-4d57-410a-891c-f89a73e1c941" containerID="d53b52eacec25d614e3559eb7a6e5775e762a8aee2911fc27018edb787495661" exitCode=0
Sep 16 20:01:23.957860 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.957703    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"
Sep 16 20:01:23.957860 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.957737    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5" event={"ID":"2db46328-4d57-410a-891c-f89a73e1c941","Type":"ContainerDied","Data":"d53b52eacec25d614e3559eb7a6e5775e762a8aee2911fc27018edb787495661"}
Sep 16 20:01:23.957860 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.957792    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5" event={"ID":"2db46328-4d57-410a-891c-f89a73e1c941","Type":"ContainerDied","Data":"3c186824d5f0864cb3752610124d7a08af04e666e7126eab9b7dfe6364764624"}
Sep 16 20:01:23.957860 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.957815    2561 scope.go:117] "RemoveContainer" containerID="d53b52eacec25d614e3559eb7a6e5775e762a8aee2911fc27018edb787495661"
Sep 16 20:01:23.968618 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.968597    2561 scope.go:117] "RemoveContainer" containerID="d53b52eacec25d614e3559eb7a6e5775e762a8aee2911fc27018edb787495661"
Sep 16 20:01:23.968902 ip-10-0-129-125 kubenswrapper[2561]: E0916 20:01:23.968880    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53b52eacec25d614e3559eb7a6e5775e762a8aee2911fc27018edb787495661\": container with ID starting with d53b52eacec25d614e3559eb7a6e5775e762a8aee2911fc27018edb787495661 not found: ID does not exist" containerID="d53b52eacec25d614e3559eb7a6e5775e762a8aee2911fc27018edb787495661"
Sep 16 20:01:23.968968 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.968914    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53b52eacec25d614e3559eb7a6e5775e762a8aee2911fc27018edb787495661"} err="failed to get container status \"d53b52eacec25d614e3559eb7a6e5775e762a8aee2911fc27018edb787495661\": rpc error: code = NotFound desc = could not find container \"d53b52eacec25d614e3559eb7a6e5775e762a8aee2911fc27018edb787495661\": container with ID starting with d53b52eacec25d614e3559eb7a6e5775e762a8aee2911fc27018edb787495661 not found: ID does not exist"
Sep 16 20:01:23.983227 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.983196    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"]
Sep 16 20:01:23.988459 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:23.988433    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-748c4bc748-8xdw5"]
Sep 16 20:01:24.735478 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:24.734782    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2db46328-4d57-410a-891c-f89a73e1c941" path="/var/lib/kubelet/pods/2db46328-4d57-410a-891c-f89a73e1c941/volumes"
Sep 16 20:01:34.420139 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:34.420088    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:37.867178 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:37.867145    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"
Sep 16 20:01:40.056802 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:40.056765    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:40.626136 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:40.626104    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:46.701771 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:46.701734    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:48.350542 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.350503    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"]
Sep 16 20:01:48.350940 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.350932    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db46328-4d57-410a-891c-f89a73e1c941" containerName="istio-proxy"
Sep 16 20:01:48.350990 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.350944    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db46328-4d57-410a-891c-f89a73e1c941" containerName="istio-proxy"
Sep 16 20:01:48.351026 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.351014    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db46328-4d57-410a-891c-f89a73e1c941" containerName="istio-proxy"
Sep 16 20:01:48.355812 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.355786    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.365709 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.365679    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"]
Sep 16 20:01:48.502572 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.502530    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.502751 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.502590    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.502751 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.502652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.502751 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.502676    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.502751 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.502697    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.502894 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.502772    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84gq5\" (UniqueName: \"kubernetes.io/projected/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-kube-api-access-84gq5\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.603792 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.603700    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.603792 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.603753    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.603792 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.603788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.604094 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.603807    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.604094 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.603827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.604094 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.604010    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84gq5\" (UniqueName: \"kubernetes.io/projected/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-kube-api-access-84gq5\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.604290 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.604268    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.604332 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.604315    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.604383 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.604362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.606361 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.606338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.606545 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.606530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.615551 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.615525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84gq5\" (UniqueName: \"kubernetes.io/projected/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-kube-api-access-84gq5\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-744pz\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.666940 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.666899    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:48.946197 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:48.946170    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"]
Sep 16 20:01:48.948156 ip-10-0-129-125 kubenswrapper[2561]: W0916 20:01:48.948126    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b526c8_a87a_4eb9_b4c8_e9cd42ca074a.slice/crio-2e25d7212ff11443ee80b7c1d35b7ae0c23df023d5c77d7786b3d297ebeea7bb WatchSource:0}: Error finding container 2e25d7212ff11443ee80b7c1d35b7ae0c23df023d5c77d7786b3d297ebeea7bb: Status 404 returned error can't find the container with id 2e25d7212ff11443ee80b7c1d35b7ae0c23df023d5c77d7786b3d297ebeea7bb
Sep 16 20:01:49.099968 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:49.099927    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz" event={"ID":"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a","Type":"ContainerStarted","Data":"f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad"}
Sep 16 20:01:49.100160 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:49.099986    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz" event={"ID":"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a","Type":"ContainerStarted","Data":"2e25d7212ff11443ee80b7c1d35b7ae0c23df023d5c77d7786b3d297ebeea7bb"}
Sep 16 20:01:49.120827 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:49.120781    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz" podStartSLOduration=1.12076572 podStartE2EDuration="1.12076572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:01:49.120424766 +0000 UTC m=+4608.923142982" watchObservedRunningTime="2026-09-16 20:01:49.12076572 +0000 UTC m=+4608.923483936"
Sep 16 20:01:49.667525 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:49.667480    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:52.013967 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.013933    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:52.060627 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.060590    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"]
Sep 16 20:01:52.065387 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.065355    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"]
Sep 16 20:01:52.117459 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.117424    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:52.125369 ip-10-0-129-125 kubenswrapper[2561]: E0916 20:01:52.125338    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:01:52.125369 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:01:52.125369 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:01:52.125369 ip-10-0-129-125 kubenswrapper[2561]:  > containerID="f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad" cmd=["/bin/sleep","15"]
Sep 16 20:01:52.125626 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.125383    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz" message="" err=<
Sep 16 20:01:52.125626 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:01:52.125626 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:01:52.125626 ip-10-0-129-125 kubenswrapper[2561]:  >
Sep 16 20:01:52.125626 ip-10-0-129-125 kubenswrapper[2561]: E0916 20:01:52.125396    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:01:52.125626 ip-10-0-129-125 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:01:52.125626 ip-10-0-129-125 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:01:52.125626 ip-10-0-129-125 kubenswrapper[2561]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz" podUID="04b526c8-a87a-4eb9-b4c8-e9cd42ca074a" containerName="main" containerID="cri-o://f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad"
Sep 16 20:01:52.125626 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.125427    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz" podUID="04b526c8-a87a-4eb9-b4c8-e9cd42ca074a" containerName="main" containerID="cri-o://f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad" gracePeriod=60
Sep 16 20:01:52.388763 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.388733    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:52.542214 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.542176    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-tmp-dir\") pod \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") "
Sep 16 20:01:52.542418 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.542228    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-home\") pod \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") "
Sep 16 20:01:52.542418 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.542255    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84gq5\" (UniqueName: \"kubernetes.io/projected/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-kube-api-access-84gq5\") pod \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") "
Sep 16 20:01:52.542418 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.542298    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-tls-certs\") pod \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") "
Sep 16 20:01:52.542418 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.542318    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-model-cache\") pod \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") "
Sep 16 20:01:52.542418 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.542344    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-dshm\") pod \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\" (UID: \"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a\") "
Sep 16 20:01:52.542686 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.542524    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-home" (OuterVolumeSpecName: "home") pod "04b526c8-a87a-4eb9-b4c8-e9cd42ca074a" (UID: "04b526c8-a87a-4eb9-b4c8-e9cd42ca074a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:52.542686 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.542583    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04b526c8-a87a-4eb9-b4c8-e9cd42ca074a" (UID: "04b526c8-a87a-4eb9-b4c8-e9cd42ca074a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:52.542686 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.542610    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-model-cache" (OuterVolumeSpecName: "model-cache") pod "04b526c8-a87a-4eb9-b4c8-e9cd42ca074a" (UID: "04b526c8-a87a-4eb9-b4c8-e9cd42ca074a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:52.542686 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.542625    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:52.544908 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.544849    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-kube-api-access-84gq5" (OuterVolumeSpecName: "kube-api-access-84gq5") pod "04b526c8-a87a-4eb9-b4c8-e9cd42ca074a" (UID: "04b526c8-a87a-4eb9-b4c8-e9cd42ca074a"). InnerVolumeSpecName "kube-api-access-84gq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:52.544908 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.544867    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04b526c8-a87a-4eb9-b4c8-e9cd42ca074a" (UID: "04b526c8-a87a-4eb9-b4c8-e9cd42ca074a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:01:52.544908 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.544866    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-dshm" (OuterVolumeSpecName: "dshm") pod "04b526c8-a87a-4eb9-b4c8-e9cd42ca074a" (UID: "04b526c8-a87a-4eb9-b4c8-e9cd42ca074a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:52.643112 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.643080    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:52.643112 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.643109    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:52.643112 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.643120    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84gq5\" (UniqueName: \"kubernetes.io/projected/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-kube-api-access-84gq5\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:52.643338 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.643129    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:52.643338 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:52.643138    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:01:53.122896 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:53.122858    2561 generic.go:356] "Generic (PLEG): container finished" podID="04b526c8-a87a-4eb9-b4c8-e9cd42ca074a" containerID="f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad" exitCode=0
Sep 16 20:01:53.123359 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:53.122918    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"
Sep 16 20:01:53.123359 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:53.122922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz" event={"ID":"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a","Type":"ContainerDied","Data":"f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad"}
Sep 16 20:01:53.123359 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:53.123043    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz" event={"ID":"04b526c8-a87a-4eb9-b4c8-e9cd42ca074a","Type":"ContainerDied","Data":"2e25d7212ff11443ee80b7c1d35b7ae0c23df023d5c77d7786b3d297ebeea7bb"}
Sep 16 20:01:53.123359 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:53.123089    2561 scope.go:117] "RemoveContainer" containerID="f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad"
Sep 16 20:01:53.133039 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:53.132873    2561 scope.go:117] "RemoveContainer" containerID="f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad"
Sep 16 20:01:53.133163 ip-10-0-129-125 kubenswrapper[2561]: E0916 20:01:53.133146    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad\": container with ID starting with f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad not found: ID does not exist" containerID="f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad"
Sep 16 20:01:53.133207 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:53.133172    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad"} err="failed to get container status \"f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad\": rpc error: code = NotFound desc = could not find container \"f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad\": container with ID starting with f6ad9945daa5dee3ef985291d7b8ca826f547fb79c726db1e33fbe6f0d3bedad not found: ID does not exist"
Sep 16 20:01:53.166950 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:53.166922    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"]
Sep 16 20:01:53.174775 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:53.174751    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-744pz"]
Sep 16 20:01:54.734271 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:54.734236    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04b526c8-a87a-4eb9-b4c8-e9cd42ca074a" path="/var/lib/kubelet/pods/04b526c8-a87a-4eb9-b4c8-e9cd42ca074a/volumes"
Sep 16 20:01:57.866968 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:01:57.866922    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh" podUID="caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:02:01.475158 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:01.475122    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:03.672441 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:03.672406    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:03.772321 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:03.772285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:07.086120 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:07.086028    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh" podUID="caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898" containerName="main" containerID="cri-o://5bb5425ba04186e9a16a85493d9c10ea08e64d4dd763936ab7759bb0bc199139" gracePeriod=45
Sep 16 20:02:07.866867 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:07.866827    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh" podUID="caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:02:17.867235 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:17.867189    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh" podUID="caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:02:17.867790 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:17.867342    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"
Sep 16 20:02:27.866777 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:27.866736    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh" podUID="caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:02:37.360650 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.360585    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"
Sep 16 20:02:37.376489 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.376454    2561 generic.go:356] "Generic (PLEG): container finished" podID="caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898" containerID="5bb5425ba04186e9a16a85493d9c10ea08e64d4dd763936ab7759bb0bc199139" exitCode=0
Sep 16 20:02:37.376617 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.376508    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"
Sep 16 20:02:37.376617 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.376540    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh" event={"ID":"caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898","Type":"ContainerDied","Data":"5bb5425ba04186e9a16a85493d9c10ea08e64d4dd763936ab7759bb0bc199139"}
Sep 16 20:02:37.376617 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.376593    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh" event={"ID":"caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898","Type":"ContainerDied","Data":"252e7d1b6f0e33ed9df5345532225a0b70bdd8ca946f0380d78148cb5ba981b7"}
Sep 16 20:02:37.376617 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.376616    2561 scope.go:117] "RemoveContainer" containerID="5bb5425ba04186e9a16a85493d9c10ea08e64d4dd763936ab7759bb0bc199139"
Sep 16 20:02:37.387579 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.387559    2561 scope.go:117] "RemoveContainer" containerID="5bb5425ba04186e9a16a85493d9c10ea08e64d4dd763936ab7759bb0bc199139"
Sep 16 20:02:37.387927 ip-10-0-129-125 kubenswrapper[2561]: E0916 20:02:37.387908    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb5425ba04186e9a16a85493d9c10ea08e64d4dd763936ab7759bb0bc199139\": container with ID starting with 5bb5425ba04186e9a16a85493d9c10ea08e64d4dd763936ab7759bb0bc199139 not found: ID does not exist" containerID="5bb5425ba04186e9a16a85493d9c10ea08e64d4dd763936ab7759bb0bc199139"
Sep 16 20:02:37.387981 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.387937    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb5425ba04186e9a16a85493d9c10ea08e64d4dd763936ab7759bb0bc199139"} err="failed to get container status \"5bb5425ba04186e9a16a85493d9c10ea08e64d4dd763936ab7759bb0bc199139\": rpc error: code = NotFound desc = could not find container \"5bb5425ba04186e9a16a85493d9c10ea08e64d4dd763936ab7759bb0bc199139\": container with ID starting with 5bb5425ba04186e9a16a85493d9c10ea08e64d4dd763936ab7759bb0bc199139 not found: ID does not exist"
Sep 16 20:02:37.437515 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.437476    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898-tls-certs\") pod \"caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898\" (UID: \"caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898\") "
Sep 16 20:02:37.437647 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.437561    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ngxq\" (UniqueName: \"kubernetes.io/projected/caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898-kube-api-access-5ngxq\") pod \"caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898\" (UID: \"caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898\") "
Sep 16 20:02:37.439696 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.439667    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898-kube-api-access-5ngxq" (OuterVolumeSpecName: "kube-api-access-5ngxq") pod "caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898" (UID: "caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898"). InnerVolumeSpecName "kube-api-access-5ngxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:02:37.439802 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.439750    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898" (UID: "caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:02:37.538997 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.538967    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:02:37.538997 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.538991    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ngxq\" (UniqueName: \"kubernetes.io/projected/caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898-kube-api-access-5ngxq\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Sep 16 20:02:37.701829 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.701796    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"]
Sep 16 20:02:37.706352 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:37.706327    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-pgndh"]
Sep 16 20:02:38.734318 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:38.734283    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898" path="/var/lib/kubelet/pods/caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898/volumes"
Sep 16 20:02:40.541140 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:40.541106    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:47.255452 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:47.255422    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:53.402355 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:53.402321    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:53.501735 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:02:53.501698    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:39.332121 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:03:39.332082    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:59.327339 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:03:59.327286    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:03.101179 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:03.101143    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:03.230310 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:03.230274    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:05.957082 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:05.957033    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:12.057155 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:12.057124    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:12.264736 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:12.264706    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:26.098498 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:26.098409    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w72vx_0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f/discovery/0.log"
Sep 16 20:04:27.173959 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:27.173930    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w72vx_0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f/discovery/0.log"
Sep 16 20:04:28.477195 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:28.477163    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-929g6_c30e896c-2d82-4ac1-a064-53912c1729b9/authorino/0.log"
Sep 16 20:04:28.515999 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:28.515973    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-x4j54_079a8e33-3974-4f19-b73e-6d170e993433/manager/0.log"
Sep 16 20:04:28.680138 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:28.680104    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-xvd7q_c3348671-0288-42bf-ae7b-14b11b51d6cd/manager/0.log"
Sep 16 20:04:28.708281 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:28.708251    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-kjvd7_ac3f1c53-ae03-45fd-b917-2c3e08951132/manager/0.log"
Sep 16 20:04:30.951290 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:30.951259    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:31.117136 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:31.117044    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:34.348842 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:34.348815    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l6stq_0c7a47e0-a2e8-4c1d-815b-18c9cecea99d/global-pull-secret-syncer/0.log"
Sep 16 20:04:34.438648 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:34.438620    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log"
Sep 16 20:04:34.545633 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:34.545601    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-125.ec2.internal_c7de017d6d245b795fdd5ece3d22237c/haproxy/0.log"
Sep 16 20:04:37.504509 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:37.504473    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:37.601844 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:37.601816    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:39.294421 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:39.294388    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-929g6_c30e896c-2d82-4ac1-a064-53912c1729b9/authorino/0.log"
Sep 16 20:04:39.424705 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:39.424657    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-x4j54_079a8e33-3974-4f19-b73e-6d170e993433/manager/0.log"
Sep 16 20:04:39.633835 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:39.633742    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-xvd7q_c3348671-0288-42bf-ae7b-14b11b51d6cd/manager/0.log"
Sep 16 20:04:39.724390 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:39.724355    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-kjvd7_ac3f1c53-ae03-45fd-b917-2c3e08951132/manager/0.log"
Sep 16 20:04:40.984314 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:40.984283    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bf491168-8cb8-4f07-81a0-d2c7a9ad0d90/alertmanager/0.log"
Sep 16 20:04:41.076025 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:41.075999    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bf491168-8cb8-4f07-81a0-d2c7a9ad0d90/config-reloader/0.log"
Sep 16 20:04:41.127549 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:41.127527    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bf491168-8cb8-4f07-81a0-d2c7a9ad0d90/kube-rbac-proxy-web/0.log"
Sep 16 20:04:41.168487 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:41.168464    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bf491168-8cb8-4f07-81a0-d2c7a9ad0d90/kube-rbac-proxy/0.log"
Sep 16 20:04:41.202973 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:41.202947    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bf491168-8cb8-4f07-81a0-d2c7a9ad0d90/kube-rbac-proxy-metric/0.log"
Sep 16 20:04:41.244093 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:41.244001    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bf491168-8cb8-4f07-81a0-d2c7a9ad0d90/prom-label-proxy/0.log"
Sep 16 20:04:41.277827 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:41.277804    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bf491168-8cb8-4f07-81a0-d2c7a9ad0d90/init-config-reloader/0.log"
Sep 16 20:04:41.328714 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:41.328689    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-f7764_08e67d03-f589-4064-862b-1fb49eff9bb2/cluster-monitoring-operator/0.log"
Sep 16 20:04:41.365505 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:41.365481    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-4hcsj_fa541550-1d07-47f5-ad29-44b4241abd11/kube-state-metrics/0.log"
Sep 16 20:04:41.463188 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:41.463162    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-4hcsj_fa541550-1d07-47f5-ad29-44b4241abd11/kube-rbac-proxy-main/0.log"
Sep 16 20:04:41.615343 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:41.615318    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-4hcsj_fa541550-1d07-47f5-ad29-44b4241abd11/kube-rbac-proxy-self/0.log"
Sep 16 20:04:41.702227 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:41.702205    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-69566c86c6-gt6sm_81a18786-abe0-4f45-80e9-040596e64ec6/metrics-server/0.log"
Sep 16 20:04:42.138854 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:42.138822    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbf95_d334d6df-acdd-48fc-af0f-484c6e26dc7c/node-exporter/0.log"
Sep 16 20:04:42.184718 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:42.184693    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbf95_d334d6df-acdd-48fc-af0f-484c6e26dc7c/kube-rbac-proxy/0.log"
Sep 16 20:04:42.230390 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:42.230359    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbf95_d334d6df-acdd-48fc-af0f-484c6e26dc7c/init-textfile/0.log"
Sep 16 20:04:42.962349 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:42.962324    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fc7857b89-4ph5b_23cfabbe-7578-4b05-b064-dff6dfbeb53a/telemeter-client/0.log"
Sep 16 20:04:43.003315 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:43.003284    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fc7857b89-4ph5b_23cfabbe-7578-4b05-b064-dff6dfbeb53a/reload/0.log"
Sep 16 20:04:43.031971 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:43.031941    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fc7857b89-4ph5b_23cfabbe-7578-4b05-b064-dff6dfbeb53a/kube-rbac-proxy/0.log"
Sep 16 20:04:43.975088 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:43.975039    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"]
Sep 16 20:04:43.975508 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:43.975443    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b526c8-a87a-4eb9-b4c8-e9cd42ca074a" containerName="main"
Sep 16 20:04:43.975508 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:43.975455    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b526c8-a87a-4eb9-b4c8-e9cd42ca074a" containerName="main"
Sep 16 20:04:43.975508 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:43.975473    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898" containerName="main"
Sep 16 20:04:43.975508 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:43.975478    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898" containerName="main"
Sep 16 20:04:43.975676 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:43.975536    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b526c8-a87a-4eb9-b4c8-e9cd42ca074a" containerName="main"
Sep 16 20:04:43.975676 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:43.975549    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf3ea1b-d1a9-4a9e-a4d7-1a8c11c84898" containerName="main"
Sep 16 20:04:43.978805 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:43.978787    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:43.982299 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:43.982261    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-srgqk\"/\"default-dockercfg-tv7gj\""
Sep 16 20:04:43.982439 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:43.982309    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-srgqk\"/\"openshift-service-ca.crt\""
Sep 16 20:04:43.983463 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:43.983448    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-srgqk\"/\"kube-root-ca.crt\""
Sep 16 20:04:43.991100 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:43.991077    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"]
Sep 16 20:04:44.111142 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.111112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90032900-7683-43a9-a6e7-41a457d592ff-proc\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.111142 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.111142    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90032900-7683-43a9-a6e7-41a457d592ff-lib-modules\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.111335 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.111187    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90032900-7683-43a9-a6e7-41a457d592ff-sys\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.111335 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.111251    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7gc9\" (UniqueName: \"kubernetes.io/projected/90032900-7683-43a9-a6e7-41a457d592ff-kube-api-access-d7gc9\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.111335 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.111293    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90032900-7683-43a9-a6e7-41a457d592ff-podres\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.212696 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.212668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90032900-7683-43a9-a6e7-41a457d592ff-podres\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.212808 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.212712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90032900-7683-43a9-a6e7-41a457d592ff-proc\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.212808 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.212735    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90032900-7683-43a9-a6e7-41a457d592ff-lib-modules\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.212808 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.212779    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90032900-7683-43a9-a6e7-41a457d592ff-sys\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.212923 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.212800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90032900-7683-43a9-a6e7-41a457d592ff-proc\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.212923 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.212821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7gc9\" (UniqueName: \"kubernetes.io/projected/90032900-7683-43a9-a6e7-41a457d592ff-kube-api-access-d7gc9\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.212923 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.212856    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90032900-7683-43a9-a6e7-41a457d592ff-podres\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.212923 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.212861    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90032900-7683-43a9-a6e7-41a457d592ff-sys\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.212923 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.212892    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90032900-7683-43a9-a6e7-41a457d592ff-lib-modules\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.223154 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.223138    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7gc9\" (UniqueName: \"kubernetes.io/projected/90032900-7683-43a9-a6e7-41a457d592ff-kube-api-access-d7gc9\") pod \"perf-node-gather-daemonset-mxzhk\" (UID: \"90032900-7683-43a9-a6e7-41a457d592ff\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.289185 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.289129    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:44.561383 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:44.561237    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"]
Sep 16 20:04:44.565127 ip-10-0-129-125 kubenswrapper[2561]: W0916 20:04:44.565094    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod90032900_7683_43a9_a6e7_41a457d592ff.slice/crio-5e7f19211ebcd9c497909b89fc7a4483260ecefc8041fe901e49a4f83291d635 WatchSource:0}: Error finding container 5e7f19211ebcd9c497909b89fc7a4483260ecefc8041fe901e49a4f83291d635: Status 404 returned error can't find the container with id 5e7f19211ebcd9c497909b89fc7a4483260ecefc8041fe901e49a4f83291d635
Sep 16 20:04:45.071161 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:45.071128    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk" event={"ID":"90032900-7683-43a9-a6e7-41a457d592ff","Type":"ContainerStarted","Data":"a55baa882ebb3fd7b96e9777ae1053a3c9ee369b5210dd300a1b9b699f163c35"}
Sep 16 20:04:45.071161 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:45.071179    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk" event={"ID":"90032900-7683-43a9-a6e7-41a457d592ff","Type":"ContainerStarted","Data":"5e7f19211ebcd9c497909b89fc7a4483260ecefc8041fe901e49a4f83291d635"}
Sep 16 20:04:45.071662 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:45.071200    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:45.091427 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:45.091378    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk" podStartSLOduration=2.091362128 podStartE2EDuration="2.091362128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:04:45.089857949 +0000 UTC m=+4784.892576167" watchObservedRunningTime="2026-09-16 20:04:45.091362128 +0000 UTC m=+4784.894080394"
Sep 16 20:04:45.338954 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:45.338878    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6f64b96cb8-xkn95_c3b46e44-354c-4b03-bc93-2aa334cf95ad/console/0.log"
Sep 16 20:04:47.050901 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:47.050870    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dk7qf_f6ce5cd3-f535-4b56-8a33-e47719901e55/dns/0.log"
Sep 16 20:04:47.097521 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:47.097492    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dk7qf_f6ce5cd3-f535-4b56-8a33-e47719901e55/kube-rbac-proxy/0.log"
Sep 16 20:04:47.431680 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:47.431591    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hm84r_0ca39171-efab-4995-a0fb-2261a074d3c9/dns-node-resolver/0.log"
Sep 16 20:04:48.065822 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:48.065777    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6f5dcf5979-vwfhs_ce3d99f9-a0f8-47fd-aa72-a98d5ce1c72f/registry/0.log"
Sep 16 20:04:48.144638 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:48.144597    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ldqrg_a5223494-bd2b-4911-ae88-8bb10b49b6f5/node-ca/0.log"
Sep 16 20:04:49.376087 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:49.376030    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w72vx_0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f/discovery/0.log"
Sep 16 20:04:50.063459 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:50.063428    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kthn2_ae49ada0-3615-4001-8c5d-4fa78bbda02d/serve-healthcheck-canary/0.log"
Sep 16 20:04:50.667979 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:50.667948    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-f2clx_84e3975d-dbd7-4c94-a16c-e61da230a8c7/insights-operator/0.log"
Sep 16 20:04:50.668946 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:50.668924    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-f2clx_84e3975d-dbd7-4c94-a16c-e61da230a8c7/insights-operator/1.log"
Sep 16 20:04:50.894116 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:50.894089    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hpnfx_9b9636af-e25e-4622-9345-ec8b6609b6ae/kube-rbac-proxy/0.log"
Sep 16 20:04:50.917647 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:50.917621    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hpnfx_9b9636af-e25e-4622-9345-ec8b6609b6ae/exporter/0.log"
Sep 16 20:04:50.940784 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:50.940668    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hpnfx_9b9636af-e25e-4622-9345-ec8b6609b6ae/extractor/0.log"
Sep 16 20:04:51.087578 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:51.087551    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-mxzhk"
Sep 16 20:04:54.133284 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:54.133253    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w72vx_0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f/discovery/0.log"
Sep 16 20:04:54.349557 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:54.349528    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvn8l_3087807d-e287-4a6d-b99c-17aa3ab3e6a5/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:55.341549 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:55.341513    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w72vx_0b3a31e0-a1e9-4a8b-bcae-23ce1167ed9f/discovery/0.log"
Sep 16 20:04:55.735004 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:55.734970    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-587fc58cc9-s9lkm_1321c471-8e55-4034-8d37-ef8e1936f7d4/manager/0.log"
Sep 16 20:04:55.766840 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:55.766776    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-br8b7_bd9604cf-01f0-4f64-9bc3-8dcc3c5def14/server/0.log"
Sep 16 20:04:57.000921 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:04:57.000875    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-zhk5r_89062e83-00cf-449d-87ad-b7e683be2bee/seaweedfs/0.log"
Sep 16 20:05:01.277200 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:05:01.277171    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"
Sep 16 20:05:01.295771 ip-10-0-129-125 kubenswrapper[2561]: I0916 20:05:01.295750    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-chgz7_5e84101c-123e-475f-ad77-fbe61af69584/ovn-acl-logging/0.log"