Jul 09 18:02:24.164347 ip-10-0-137-79 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 09 18:02:24.164358 ip-10-0-137-79 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 09 18:02:24.164368 ip-10-0-137-79 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 09 18:02:24.164704 ip-10-0-137-79 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 09 18:02:35.433427 ip-10-0-137-79 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 09 18:02:35.433448 ip-10-0-137-79 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3d5cd3a3b60b4dbc94556940ec9445b5 --
Jul 09 18:04:36.692145 ip-10-0-137-79 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 18:04:37.111323 ip-10-0-137-79 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:37.111323 ip-10-0-137-79 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 18:04:37.111323 ip-10-0-137-79 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:37.111323 ip-10-0-137-79 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 18:04:37.111323 ip-10-0-137-79 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:37.112934 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.112812    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 18:04:37.119185 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119146    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:37.119185 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119178    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:37.119185 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119182    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:37.119185 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119185    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:37.119185 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119189    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:37.119185 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119191    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:37.119185 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119194    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:37.119185 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119197    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:37.119185 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119199    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:37.119185 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119202    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119208    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119212    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119216    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119219    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119222    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119225    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119227    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119230    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119232    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119236    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119238    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119242    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119248    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119252    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119256    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119259    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119262    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119264    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:37.119551 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119267    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119270    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119276    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119279    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119282    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119285    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119296    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119300    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119303    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119307    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119309    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119312    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119315    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119317    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119319    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119325    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119328    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119331    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119335    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:37.120011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119337    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119340    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119343    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119346    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119349    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119353    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119355    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119358    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119361    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119363    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119366    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119368    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119371    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119373    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119376    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119378    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119381    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119384    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119386    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119389    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:37.120450 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119392    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119395    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119397    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119400    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119402    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119405    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119407    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119410    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119414    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119417    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119420    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119422    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119425    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119427    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119430    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119432    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119436    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119439    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119441    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119444    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:37.120966 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119446    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119449    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119452    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119454    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119457    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119459    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119462    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119464    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119467    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119469    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119471    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.119474    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120239    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120246    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120250    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120255    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120259    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120262    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120265    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:37.121460 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120268    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120271    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120273    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120276    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120279    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120281    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120284    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120286    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120289    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120292    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120295    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120298    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120300    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120303    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120305    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120308    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120310    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120313    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120315    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120318    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:37.121968 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120321    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120323    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120326    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120329    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120331    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120334    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120337    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120339    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120342    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120346    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120349    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120351    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120354    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120356    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120359    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120361    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120364    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120366    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120369    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120372    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:37.122443 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120374    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120380    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120382    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120385    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120388    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120390    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120392    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120395    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120399    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120402    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120405    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120408    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120410    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120412    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120415    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120418    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120420    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120424    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120428    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:37.122969 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120431    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120434    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120437    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120439    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120442    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120467    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120472    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120475    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120478    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120480    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120483    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120486    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120489    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120491    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120495    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120497    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120500    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120503    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120506    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:37.123548 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120508    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120511    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120514    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120516    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120519    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120521    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120524    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120527    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120529    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120532    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120535    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120538    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120540    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120543    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120969    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120978    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120981    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120984    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120986    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:37.124025 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120989    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120992    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120994    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.120997    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121000    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121003    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121005    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121008    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121010    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121013    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121016    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121019    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121022    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121024    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121026    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121031    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121034    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121037    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121040    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:37.124477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121042    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121045    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121047    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121050    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121052    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121055    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121057    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121060    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121062    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121066    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121069    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121071    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121074    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121076    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121079    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121081    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121084    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121087    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121089    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121092    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:37.124964 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121094    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121097    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121099    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121102    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121104    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121107    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121110    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121112    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121116    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121119    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121121    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121124    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121126    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121129    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121132    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121134    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121136    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121139    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121143    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121146    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:37.125477 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121149    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121154    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121157    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121160    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121163    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121166    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121168    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121171    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121173    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121176    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121178    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121181    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121184    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121187    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121189    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121192    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121194    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121197    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121200    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121203    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:37.125970 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121207    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121209    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121212    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121216    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121218    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121221    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121225    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121229    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121232    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121235    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121238    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121240    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121244    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121249    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121253    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121330    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121337    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121344    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121349    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121353    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 18:04:37.126444 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121356    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121361    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121365    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121368    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121372    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121375    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121379    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121382    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121385    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121388    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121391    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121394    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121399    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121406    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121409    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121412    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121415    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121419    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121423    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121426    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121429    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121432    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121436    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121439    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 18:04:37.126933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121442    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121446    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121449    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121454    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121457    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121460    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121463    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121466    2578 flags.go:64] FLAG: --enable-server="true"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121469    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121474    2578 flags.go:64] FLAG: --event-burst="100"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121478    2578 flags.go:64] FLAG: --event-qps="50"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121480    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121484    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121487    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121491    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121494    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121497    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121500    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121503    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121506    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121511    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121514    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121517    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121520    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121523    2578 flags.go:64] FLAG: --feature-gates=""
Jul 09 18:04:37.127542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121526    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121529    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121533    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121536    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121539    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121542    2578 flags.go:64] FLAG: --help="false"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121545    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121548    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121551    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121554    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121558    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121561    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121564    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121567    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121570    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121572    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121575    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121578    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121581    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121583    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121586    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121589    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121592    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121594    2578 flags.go:64] FLAG: --lock-file=""
Jul 09 18:04:37.128207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121597    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121600    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121602    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121608    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121612    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121615    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121618    2578 flags.go:64] FLAG: --logging-format="text"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121620    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121624    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121626    2578 flags.go:64] FLAG: --manifest-url=""
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121629    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121633    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121637    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121641    2578 flags.go:64] FLAG: --max-pods="110"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121644    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121647    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121650    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121653    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121656    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121659    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121662    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121670    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121673    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121676    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 18:04:37.128825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121679    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121682    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121688    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121691    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121695    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121698    2578 flags.go:64] FLAG: --port="10250"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121701    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121704    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f1ee29256bc59e07"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121707    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121710    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121713    2578 flags.go:64] FLAG: --register-node="true"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121716    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121720    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121723    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121726    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121729    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121732    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121736    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121739    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121742    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121744    2578 flags.go:64] FLAG: --runonce="false"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121747    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121750    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121753    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121756    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 18:04:37.129403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121759    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121762    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121765    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121768    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121770    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121773    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121776    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121779    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121782    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121785    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121788    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121805    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121808    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121811    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121815    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121818    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121820    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121823    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121826    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121830    2578 flags.go:64] FLAG: --v="2"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121834    2578 flags.go:64] FLAG: --version="false"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121839    2578 flags.go:64] FLAG: --vmodule=""
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121842    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.121846    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 18:04:37.130024 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121963    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121969    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121973    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121976    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121979    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121983    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121985    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121988    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121991    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121994    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121996    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.121999    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122001    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122003    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122006    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122008    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122011    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122013    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122018    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:37.130595 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122020    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122023    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122026    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122028    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122031    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122033    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122036    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122039    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122041    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122044    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122048    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122052    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122054    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122057    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122059    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122063    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122066    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122069    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122071    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122074    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:37.131186 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122076    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122079    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122081    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122084    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122086    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122089    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122092    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122094    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122097    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122099    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122102    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122106    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122108    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122111    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122113    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122117    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122121    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122123    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122126    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122129    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:37.131667 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122132    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122135    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122138    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122140    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122143    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122146    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122148    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122153    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122156    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122158    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122161    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122164    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122166    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122169    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122171    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122174    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122176    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122179    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122182    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122184    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:37.132204 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122186    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122189    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122191    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122195    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122198    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122200    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122203    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122205    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122208    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122211    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122213    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122216    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122218    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122221    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122223    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122226    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122228    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122231    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122233    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:37.132694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.122238    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.122244    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.129083    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.129103    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.129920    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.129929    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.129995    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130002    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130005    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130008    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130011    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130014    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130017    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130020    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130023    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:37.133181 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130026    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130028    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130031    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130034    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130037    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130040    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130043    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130046    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130048    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130051    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130053    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130056    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130059    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130063    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130067    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130070    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130073    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130076    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130078    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130081    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:37.133540 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130085    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130089    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130093    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130096    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130098    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130101    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130103    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130106    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130109    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130111    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130114    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130116    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130119    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130122    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130125    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130127    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130130    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130133    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130135    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:37.134049 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130138    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130140    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130143    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130145    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130148    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130150    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130153    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130155    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130158    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130160    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130163    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130165    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130168    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130171    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130174    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130177    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130181    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130185    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130187    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130190    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:37.134514 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130193    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130196    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130199    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130201    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130204    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130206    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130209    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130212    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130215    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130217    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130220    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130223    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130225    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130228    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130230    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130233    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130236    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130238    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130241    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:37.135011 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130243    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130246    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130249    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130251    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130254    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130256    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130260    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130263    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130266    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130269    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130271    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130273    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.130279    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130398    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130404    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:37.135469 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130408    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130411    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130414    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130417    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130419    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130422    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130426    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130430    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130433    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130435    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130439    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130442    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130445    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130448    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130450    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130453    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130455    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130457    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130460    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:37.135917 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130463    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130465    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130467    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130470    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130472    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130475    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130478    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130481    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130484    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130486    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130489    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130492    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130494    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130497    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130500    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130502    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130505    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130507    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130510    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130513    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:37.136380 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130516    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130518    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130521    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130523    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130526    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130529    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130531    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130534    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130536    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130539    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130541    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130543    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130546    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130548    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130551    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130553    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130556    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130558    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130561    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130564    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130567    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:37.136881 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130571    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130573    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130576    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130578    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130582    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130584    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130587    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130589    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130592    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130595    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130598    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130601    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130603    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130606    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130609    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130613    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130616    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130619    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130621    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:37.137392 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130624    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130626    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130629    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130631    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130634    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130636    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130639    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130642    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130644    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130647    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130649    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130652    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130654    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130658    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130660    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130663    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130665    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:37.137883 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:37.130668    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:37.138398 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.130673    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:37.138398 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.130830    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 18:04:37.138398 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.133196    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 18:04:37.138398 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.134877    2578 server.go:1019] "Starting client certificate rotation"
Jul 09 18:04:37.138398 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.134977    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:37.138398 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.135015    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:37.138398 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.137561    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:37.146294 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.146269    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:37.153470 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.153446    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 09 18:04:37.157349 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.157331    2578 log.go:25] "Validated CRI v1 image API"
Jul 09 18:04:37.158198 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.158168    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 18:04:37.160882 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.160861    2578 fs.go:135] Filesystem UUIDs: map[38d94d12-71db-4dff-bfdd-eb9ff035052f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9098ced7-6e5f-472f-ad8a-54c9a47221e4:/dev/nvme0n1p3]
Jul 09 18:04:37.160980 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.160881    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 18:04:37.166002 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.165877    2578 manager.go:216] Machine: {Timestamp:2026-07-09 18:04:37.164611776 +0000 UTC m=+0.351589536 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3113378 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bd50d2a43746349898c8d9514a595 SystemUUID:ec2bd50d-2a43-7463-4989-8c8d9514a595 BootID:3d5cd3a3-b60b-4dbc-9455-6940ec9445b5 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b9:75:09:b3:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b9:75:09:b3:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:09:db:ea:b5:52 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 18:04:37.166002 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.166002    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 18:04:37.166114 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.166096    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 18:04:37.169160 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.169126    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 18:04:37.169311 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.169166    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-79.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 18:04:37.170450 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.170439    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 18:04:37.170484 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.170453    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 18:04:37.170484 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.170461    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:37.171228 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.171214    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:37.172552 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.172541    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 18:04:37.175496 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.175483    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:37.175688 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.175678    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 18:04:37.175721 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.175711    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 18:04:37.175757 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.175722    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 18:04:37.175757 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.175742    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 18:04:37.175757 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.175756    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 09 18:04:37.175869 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.175770    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 18:04:37.175989 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.175971    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:37.176277 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.176264    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:37.176313 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.176286    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:37.178534 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.178512    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 18:04:37.180931 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.180909    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 18:04:37.181034 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.180957    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 18:04:37.181186 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.181172    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 18:04:37.181227 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.181196    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 18:04:37.181227 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.181206    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 18:04:37.181227 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.181216    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 18:04:37.181356 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.181236    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 18:04:37.181356 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.181245    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 18:04:37.182080 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.182065    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 18:04:37.182131 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.182084    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 18:04:37.182131 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.182099    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 18:04:37.182131 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.182109    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 18:04:37.182131 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.182124    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 18:04:37.182274 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.182140    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 18:04:37.183155 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.183138    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 18:04:37.183234 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.183218    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 18:04:37.186836 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.186824    2578 server.go:1268] "Started kubelet"
Jul 09 18:04:37.187078 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.186921    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 18:04:37.187078 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.186911    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 18:04:37.187078 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.186980    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 18:04:37.187706 ip-10-0-137-79 systemd[1]: Started Kubernetes Kubelet.
Jul 09 18:04:37.188456 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.188417    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 18:04:37.188852 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.188838    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 18:04:37.191678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.191650    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-79.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 18:04:37.191771 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.191691    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 18:04:37.191771 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.191748    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 18:04:37.194013 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.193993    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 18:04:37.194124 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.194038    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:37.194960 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.194302    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-79.ec2.internal\" not found"
Jul 09 18:04:37.194960 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.194862    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 18:04:37.194960 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.194876    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 18:04:37.194960 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.194881    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 18:04:37.195172 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.195019    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 18:04:37.195172 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.195026    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 18:04:37.195172 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.195142    2578 factory.go:55] Registering systemd factory
Jul 09 18:04:37.195172 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.195156    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 09 18:04:37.195754 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.195740    2578 factory.go:153] Registering CRI-O factory
Jul 09 18:04:37.195863 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.195853    2578 factory.go:223] Registration of the crio container factory successfully
Jul 09 18:04:37.195994 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.195985    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 18:04:37.196119 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.196110    2578 factory.go:103] Registering Raw factory
Jul 09 18:04:37.196187 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.196181    2578 manager.go:1196] Started watching for new ooms in manager
Jul 09 18:04:37.197506 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.197328    2578 manager.go:349] Starting recovery of all containers
Jul 09 18:04:37.198302 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.198277    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 18:04:37.198518 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.198488    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 18:04:37.199855 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.199814    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 18:04:37.200699 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.198736    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-79.ec2.internal.18c0b1312be6d0ee  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-79.ec2.internal,UID:ip-10-0-137-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-79.ec2.internal,},FirstTimestamp:2026-07-09 18:04:37.186777326 +0000 UTC m=+0.373755084,LastTimestamp:2026-07-09 18:04:37.186777326 +0000 UTC m=+0.373755084,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-79.ec2.internal,}"
Jul 09 18:04:37.208627 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.208329    2578 manager.go:354] Recovery completed
Jul 09 18:04:37.214219 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.214205    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:37.216253 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.216226    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:37.216335 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.216276    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:37.216335 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.216295    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:37.216896 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.216877    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 18:04:37.216896 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.216897    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:37.216987 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.216896    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 18:04:37.216987 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.216962    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:37.217597 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.217582    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:37.217669 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.217607    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:37.217669 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.217620    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:37.218056 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.218044    2578 policy_none.go:49] "None policy: Start"
Jul 09 18:04:37.218089 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.218061    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 18:04:37.218089 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.218071    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 18:04:37.218977 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.218902    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-79.ec2.internal.18c0b1312da89cd2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-79.ec2.internal,UID:ip-10-0-137-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-79.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-79.ec2.internal,},FirstTimestamp:2026-07-09 18:04:37.216255186 +0000 UTC m=+0.403232951,LastTimestamp:2026-07-09 18:04:37.216255186 +0000 UTC m=+0.403232951,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-79.ec2.internal,}"
Jul 09 18:04:37.219315 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.219303    2578 policy_none.go:47] "Start"
Jul 09 18:04:37.235012 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.234927    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-79.ec2.internal.18c0b1312da910c9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-79.ec2.internal,UID:ip-10-0-137-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-79.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-79.ec2.internal,},FirstTimestamp:2026-07-09 18:04:37.216284873 +0000 UTC m=+0.403262634,LastTimestamp:2026-07-09 18:04:37.216284873 +0000 UTC m=+0.403262634,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-79.ec2.internal,}"
Jul 09 18:04:37.248811 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.248728    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-79.ec2.internal.18c0b1312da9521c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-79.ec2.internal,UID:ip-10-0-137-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-79.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-79.ec2.internal,},FirstTimestamp:2026-07-09 18:04:37.216301596 +0000 UTC m=+0.403279356,LastTimestamp:2026-07-09 18:04:37.216301596 +0000 UTC m=+0.403279356,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-79.ec2.internal,}"
Jul 09 18:04:37.259541 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.259518    2578 manager.go:337] "Starting Device Plugin manager"
Jul 09 18:04:37.259646 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.259571    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 18:04:37.259646 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.259583    2578 server.go:87] "Starting device plugin registration server"
Jul 09 18:04:37.259954 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.259875    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-79.ec2.internal.18c0b1312da89cd2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-79.ec2.internal.18c0b1312da89cd2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-79.ec2.internal,UID:ip-10-0-137-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-79.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-79.ec2.internal,},FirstTimestamp:2026-07-09 18:04:37.216255186 +0000 UTC m=+0.403232951,LastTimestamp:2026-07-09 18:04:37.217595255 +0000 UTC m=+0.404573013,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-79.ec2.internal,}"
Jul 09 18:04:37.260045 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.259940    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 18:04:37.260090 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.260036    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 18:04:37.260149 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.260132    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 18:04:37.260263 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.260239    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 18:04:37.260263 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.260260    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 18:04:37.260807 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.260757    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 18:04:37.260894 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.260837    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-79.ec2.internal\" not found"
Jul 09 18:04:37.271714 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.271634    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-79.ec2.internal.18c0b1312da910c9\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-79.ec2.internal.18c0b1312da910c9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-79.ec2.internal,UID:ip-10-0-137-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-79.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-79.ec2.internal,},FirstTimestamp:2026-07-09 18:04:37.216284873 +0000 UTC m=+0.403262634,LastTimestamp:2026-07-09 18:04:37.217612297 +0000 UTC m=+0.404590061,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-79.ec2.internal,}"
Jul 09 18:04:37.284489 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.284383    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-79.ec2.internal.18c0b1312da9521c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-79.ec2.internal.18c0b1312da9521c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-79.ec2.internal,UID:ip-10-0-137-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-79.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-79.ec2.internal,},FirstTimestamp:2026-07-09 18:04:37.216301596 +0000 UTC m=+0.403279356,LastTimestamp:2026-07-09 18:04:37.217626391 +0000 UTC m=+0.404604150,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-79.ec2.internal,}"
Jul 09 18:04:37.297718 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.297642    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-79.ec2.internal.18c0b13130636f7a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-79.ec2.internal,UID:ip-10-0-137-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-137-79.ec2.internal,},FirstTimestamp:2026-07-09 18:04:37.262053242 +0000 UTC m=+0.449030988,LastTimestamp:2026-07-09 18:04:37.262053242 +0000 UTC m=+0.449030988,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-79.ec2.internal,}"
Jul 09 18:04:37.299312 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.299293    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zs5v9"
Jul 09 18:04:37.307269 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.307249    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zs5v9"
Jul 09 18:04:37.360896 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.360857    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:37.362104 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.362059    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:37.362104 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.362090    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:37.362104 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.362104    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:37.362235 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.362131    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.381758 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.381723    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.381758 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.381757    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-79.ec2.internal\": node \"ip-10-0-137-79.ec2.internal\" not found"
Jul 09 18:04:37.395650 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.395609    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 18:04:37.396914 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.396894    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 18:04:37.397022 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.396942    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 18:04:37.397022 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.396975    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 18:04:37.397022 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.397021    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 18:04:37.399706 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.399680    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:37.414003 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.413980    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-79.ec2.internal\" not found"
Jul 09 18:04:37.497752 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.497137    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal"]
Jul 09 18:04:37.497861 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.497811    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:37.498888 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.498871    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:37.498939 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.498906    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:37.498939 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.498921    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:37.500262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.500249    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:37.500380 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.500364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.500414 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.500396    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:37.501099 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.501082    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:37.501171 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.501117    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:37.501171 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.501128    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:37.501231 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.501082    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:37.501231 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.501192    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:37.501231 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.501204    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:37.502346 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.502332    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.502394 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.502362    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:37.503188 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.503171    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:37.503279 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.503202    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:37.503279 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.503214    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:37.514962 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.514931    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-79.ec2.internal\" not found"
Jul 09 18:04:37.526703 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.526679    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-79.ec2.internal\" not found" node="ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.531110 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.531089    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-79.ec2.internal\" not found" node="ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.597123 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.597092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ecb5e10e9a5b0d3e70008ee6ce282ea9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal\" (UID: \"ecb5e10e9a5b0d3e70008ee6ce282ea9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.597123 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.597124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecb5e10e9a5b0d3e70008ee6ce282ea9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal\" (UID: \"ecb5e10e9a5b0d3e70008ee6ce282ea9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.597386 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.597145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb2f2f9ee8258d42945cd1eb17f2050c-config\") pod \"kube-apiserver-proxy-ip-10-0-137-79.ec2.internal\" (UID: \"fb2f2f9ee8258d42945cd1eb17f2050c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.615638 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.615582    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-79.ec2.internal\" not found"
Jul 09 18:04:37.697848 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.697818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ecb5e10e9a5b0d3e70008ee6ce282ea9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal\" (UID: \"ecb5e10e9a5b0d3e70008ee6ce282ea9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.697940 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.697865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecb5e10e9a5b0d3e70008ee6ce282ea9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal\" (UID: \"ecb5e10e9a5b0d3e70008ee6ce282ea9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.697940 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.697881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ecb5e10e9a5b0d3e70008ee6ce282ea9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal\" (UID: \"ecb5e10e9a5b0d3e70008ee6ce282ea9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.697940 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.697904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb2f2f9ee8258d42945cd1eb17f2050c-config\") pod \"kube-apiserver-proxy-ip-10-0-137-79.ec2.internal\" (UID: \"fb2f2f9ee8258d42945cd1eb17f2050c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.697940 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.697884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb2f2f9ee8258d42945cd1eb17f2050c-config\") pod \"kube-apiserver-proxy-ip-10-0-137-79.ec2.internal\" (UID: \"fb2f2f9ee8258d42945cd1eb17f2050c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.697940 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.697927    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecb5e10e9a5b0d3e70008ee6ce282ea9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal\" (UID: \"ecb5e10e9a5b0d3e70008ee6ce282ea9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.715947 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.715918    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-79.ec2.internal\" not found"
Jul 09 18:04:37.816446 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.816410    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-79.ec2.internal\" not found"
Jul 09 18:04:37.828617 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.828583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.834386 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:37.834368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:37.917010 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:37.916912    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-79.ec2.internal\" not found"
Jul 09 18:04:38.017457 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:38.017416    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-79.ec2.internal\" not found"
Jul 09 18:04:38.118008 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:38.117978    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-79.ec2.internal\" not found"
Jul 09 18:04:38.134129 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.134098    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 18:04:38.134309 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.134287    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:04:38.218852 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:38.218744    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-79.ec2.internal\" not found"
Jul 09 18:04:38.310228 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.310173    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:59:37 +0000 UTC" deadline="2028-02-17 02:12:57.09277358 +0000 UTC"
Jul 09 18:04:38.310228 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.310216    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14096h8m18.782560859s"
Jul 09 18:04:38.319360 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:38.319327    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-79.ec2.internal\" not found"
Jul 09 18:04:38.417331 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.417301    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:38.418526 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.418507    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:38.418632 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.418539    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:38.418632 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.418549    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:38.420414 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:38.420344    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-79.ec2.internal\" not found"
Jul 09 18:04:38.453562 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:38.453518    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb5e10e9a5b0d3e70008ee6ce282ea9.slice/crio-43cfb95b34d64a7264595b898c273cfa7851c2f07f51e2b19bf471dffac3cd74 WatchSource:0}: Error finding container 43cfb95b34d64a7264595b898c273cfa7851c2f07f51e2b19bf471dffac3cd74: Status 404 returned error can't find the container with id 43cfb95b34d64a7264595b898c273cfa7851c2f07f51e2b19bf471dffac3cd74
Jul 09 18:04:38.453782 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:38.453762    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2f2f9ee8258d42945cd1eb17f2050c.slice/crio-cf6a313f2856ee669b124b0f1a3162df6be60fe6e9e41ed1f2f74a9b2241e13f WatchSource:0}: Error finding container cf6a313f2856ee669b124b0f1a3162df6be60fe6e9e41ed1f2f74a9b2241e13f: Status 404 returned error can't find the container with id cf6a313f2856ee669b124b0f1a3162df6be60fe6e9e41ed1f2f74a9b2241e13f
Jul 09 18:04:38.459858 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.459837    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:04:38.460277 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.460248    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:38.494654 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.494559    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:38.507456 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.507429    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:04:38.509047 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.509031    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal"
Jul 09 18:04:38.528553 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.528524    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:04:38.717021 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.716985    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:38.781778 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:38.781695    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:39.176990 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.176899    2578 apiserver.go:52] "Watching apiserver"
Jul 09 18:04:39.185129 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.185095    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 18:04:39.186644 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.186611    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5kvp7","kube-system/konnectivity-agent-42brv","kube-system/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qpm9g","openshift-image-registry/node-ca-klwxw","openshift-multus/multus-additional-cni-plugins-8jmw8","openshift-network-operator/iptables-alerter-5vlnq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9","openshift-dns/node-resolver-d7pqh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal","openshift-multus/multus-8qxqd","openshift-multus/network-metrics-daemon-kpcpn","openshift-network-diagnostics/network-check-target-f6494","openshift-ovn-kubernetes/ovnkube-node-nlk4c"]
Jul 09 18:04:39.188586 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.188435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5vlnq"
Jul 09 18:04:39.189682 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.189646    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:39.189832 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.189793    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:04:39.191079 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.191053    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.191592 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.191567    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 18:04:39.191737 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.191630    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j5g4d\""
Jul 09 18:04:39.191943 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.191894    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:39.192459 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.192440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.192749 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.192732    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:39.195558 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.195153    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 18:04:39.195558 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.195224    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qwp2m\""
Jul 09 18:04:39.195558 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.195489    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:39.195755 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.195634    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 18:04:39.195958 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.195937    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-klwxw"
Jul 09 18:04:39.196692 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.196670    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:39.197107 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.196990    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.197107 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.197048    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 18:04:39.197107 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.197085    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 18:04:39.197426 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.197375    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8hzgk\""
Jul 09 18:04:39.197918 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.197549    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 18:04:39.197918 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.197702    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:39.198071 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.197977    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 18:04:39.198262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.198242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d7pqh"
Jul 09 18:04:39.199826 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.199640    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:39.199826 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.199709    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:04:39.202761 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.202185    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-42brv"
Jul 09 18:04:39.206010 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.205422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.206263 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.206243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-node-log\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.206351 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.206284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-cni-bin\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.206413 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.206397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-cni-netd\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.206469 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.206442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-sysctl-d\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.206528 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.206470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa299791-af91-4d8a-91f2-0b87b0a9732e-tmp\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.206528 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.206509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.206625 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.206588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-modprobe-d\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.206677 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.206629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-sysctl-conf\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.206753 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.206734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-ovnkube-config\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.206791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-ovn-node-metrics-cert\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207357    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2b7t\" (UniqueName: \"kubernetes.io/projected/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-kube-api-access-d2b7t\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-var-lib-kubelet\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207435    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c290aebc-bdd2-47e7-a307-f00b1df88b5f-host\") pod \"node-ca-klwxw\" (UID: \"c290aebc-bdd2-47e7-a307-f00b1df88b5f\") " pod="openshift-image-registry/node-ca-klwxw"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj4cx\" (UniqueName: \"kubernetes.io/projected/c290aebc-bdd2-47e7-a307-f00b1df88b5f-kube-api-access-rj4cx\") pod \"node-ca-klwxw\" (UID: \"c290aebc-bdd2-47e7-a307-f00b1df88b5f\") " pod="openshift-image-registry/node-ca-klwxw"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f4a275b-51e0-4ef7-ba98-196b5589a2d8-iptables-alerter-script\") pod \"iptables-alerter-5vlnq\" (UID: \"9f4a275b-51e0-4ef7-ba98-196b5589a2d8\") " pod="openshift-network-operator/iptables-alerter-5vlnq"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-run-systemd\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-log-socket\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-run-ovn-kubernetes\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-ovnkube-script-lib\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-run\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c290aebc-bdd2-47e7-a307-f00b1df88b5f-serviceca\") pod \"node-ca-klwxw\" (UID: \"c290aebc-bdd2-47e7-a307-f00b1df88b5f\") " pod="openshift-image-registry/node-ca-klwxw"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207724    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-host\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-tuned\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw76z\" (UniqueName: \"kubernetes.io/projected/9f4a275b-51e0-4ef7-ba98-196b5589a2d8-kube-api-access-sw76z\") pod \"iptables-alerter-5vlnq\" (UID: \"9f4a275b-51e0-4ef7-ba98-196b5589a2d8\") " pod="openshift-network-operator/iptables-alerter-5vlnq"
Jul 09 18:04:39.208058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j62gd\" (UniqueName: \"kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd\") pod \"network-check-target-f6494\" (UID: \"62eb9ea7-9552-4c47-a202-adbba7cd45fa\") " pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207886    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-kubelet\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-var-lib-openvswitch\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.207980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-sys\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.208012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-lib-modules\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.208053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-slash\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.208085    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-etc-openvswitch\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.208111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-env-overrides\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.208150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-kubernetes\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.208182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-systemd\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.208224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54lqg\" (UniqueName: \"kubernetes.io/projected/aa299791-af91-4d8a-91f2-0b87b0a9732e-kube-api-access-54lqg\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.208261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-systemd-units\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.208286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-run-ovn\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.208317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-sysconfig\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.208340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f4a275b-51e0-4ef7-ba98-196b5589a2d8-host-slash\") pod \"iptables-alerter-5vlnq\" (UID: \"9f4a275b-51e0-4ef7-ba98-196b5589a2d8\") " pod="openshift-network-operator/iptables-alerter-5vlnq"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.208364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-run-netns\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.208388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-run-openvswitch\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.208906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.208593    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.211386 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.211334    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:39.211485 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.211412    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:04:39.221650 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.221263    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 18:04:39.221650 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.221359    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 18:04:39.222016 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.221888    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 18:04:39.222016 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.221930    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 18:04:39.222213 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.222193    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 18:04:39.222280 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.222252    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 18:04:39.222442 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.222364    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 18:04:39.222627 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.222602    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 18:04:39.222721 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.222684    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 18:04:39.222721 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.222610    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 18:04:39.222829 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.222782    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 18:04:39.222901 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.222887    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 18:04:39.223027 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.223011    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 18:04:39.223078 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.223044    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 18:04:39.223164 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.223149    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zhvng\""
Jul 09 18:04:39.223254 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.223223    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pjxsz\""
Jul 09 18:04:39.223317 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.223288    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 18:04:39.223420 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.223400    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q27q5\""
Jul 09 18:04:39.223477 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.223434    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k4bhf\""
Jul 09 18:04:39.223531 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.223486    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 18:04:39.223607 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.223591    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2tx24\""
Jul 09 18:04:39.223725 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.223706    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cq5v2\""
Jul 09 18:04:39.250542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.250437    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:39.279565 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.279527    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dknmb"
Jul 09 18:04:39.291843 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.291795    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dknmb"
Jul 09 18:04:39.296841 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.296736    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 18:04:39.309040 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-run-k8s-cni-cncf-io\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.309239 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-run-netns\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.309239 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-cni-netd\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.309239 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa299791-af91-4d8a-91f2-0b87b0a9732e-tmp\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.309239 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-sysctl-conf\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.309239 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-multus-conf-dir\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.309239 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-ovnkube-config\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.309239 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-var-lib-kubelet\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.309621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c290aebc-bdd2-47e7-a307-f00b1df88b5f-host\") pod \"node-ca-klwxw\" (UID: \"c290aebc-bdd2-47e7-a307-f00b1df88b5f\") " pod="openshift-image-registry/node-ca-klwxw"
Jul 09 18:04:39.309621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4cx\" (UniqueName: \"kubernetes.io/projected/c290aebc-bdd2-47e7-a307-f00b1df88b5f-kube-api-access-rj4cx\") pod \"node-ca-klwxw\" (UID: \"c290aebc-bdd2-47e7-a307-f00b1df88b5f\") " pod="openshift-image-registry/node-ca-klwxw"
Jul 09 18:04:39.309621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r6rv\" (UniqueName: \"kubernetes.io/projected/8a408701-9b8b-48a1-b843-165cb5758437-kube-api-access-2r6rv\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.309621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-registration-dir\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.309621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-sys-fs\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.309621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ccfb877-5ee9-40fa-a28d-8310e8922463-tmp-dir\") pod \"node-resolver-d7pqh\" (UID: \"9ccfb877-5ee9-40fa-a28d-8310e8922463\") " pod="openshift-dns/node-resolver-d7pqh"
Jul 09 18:04:39.309621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-log-socket\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.309621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-ovnkube-script-lib\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.309621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c290aebc-bdd2-47e7-a307-f00b1df88b5f-serviceca\") pod \"node-ca-klwxw\" (UID: \"c290aebc-bdd2-47e7-a307-f00b1df88b5f\") " pod="openshift-image-registry/node-ca-klwxw"
Jul 09 18:04:39.309621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-var-lib-cni-bin\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.309621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-var-lib-cni-multus\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.309621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-run-multus-certs\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.309621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9f223b1-0765-46bb-b8af-136743836d8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.309621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ccfb877-5ee9-40fa-a28d-8310e8922463-hosts-file\") pod \"node-resolver-d7pqh\" (UID: \"9ccfb877-5ee9-40fa-a28d-8310e8922463\") " pod="openshift-dns/node-resolver-d7pqh"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-host\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-tuned\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw76z\" (UniqueName: \"kubernetes.io/projected/9f4a275b-51e0-4ef7-ba98-196b5589a2d8-kube-api-access-sw76z\") pod \"iptables-alerter-5vlnq\" (UID: \"9f4a275b-51e0-4ef7-ba98-196b5589a2d8\") " pod="openshift-network-operator/iptables-alerter-5vlnq"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309738    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-var-lib-kubelet\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwc6n\" (UniqueName: \"kubernetes.io/projected/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-kube-api-access-kwc6n\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-kubelet\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-lib-modules\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-hostroot\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9f223b1-0765-46bb-b8af-136743836d8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-kubernetes\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-systemd\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.309974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54lqg\" (UniqueName: \"kubernetes.io/projected/aa299791-af91-4d8a-91f2-0b87b0a9732e-kube-api-access-54lqg\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310004    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e441cb9-53b1-41e0-8f86-c0d30c8dd55f-agent-certs\") pod \"konnectivity-agent-42brv\" (UID: \"8e441cb9-53b1-41e0-8f86-c0d30c8dd55f\") " pod="kube-system/konnectivity-agent-42brv"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9f223b1-0765-46bb-b8af-136743836d8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-systemd-units\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-run-ovn\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.310390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9f223b1-0765-46bb-b8af-136743836d8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-run-openvswitch\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-node-log\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-cni-bin\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-sysctl-d\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3da572d-acbd-4c3c-a693-c0eac330f719-dbus\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-modprobe-d\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2tnd\" (UniqueName: \"kubernetes.io/projected/9ccfb877-5ee9-40fa-a28d-8310e8922463-kube-api-access-z2tnd\") pod \"node-resolver-d7pqh\" (UID: \"9ccfb877-5ee9-40fa-a28d-8310e8922463\") " pod="openshift-dns/node-resolver-d7pqh"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-ovn-node-metrics-cert\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2b7t\" (UniqueName: \"kubernetes.io/projected/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-kube-api-access-d2b7t\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f4a275b-51e0-4ef7-ba98-196b5589a2d8-iptables-alerter-script\") pod \"iptables-alerter-5vlnq\" (UID: \"9f4a275b-51e0-4ef7-ba98-196b5589a2d8\") " pod="openshift-network-operator/iptables-alerter-5vlnq"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbx4z\" (UniqueName: \"kubernetes.io/projected/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-kube-api-access-jbx4z\") pod \"network-metrics-daemon-kpcpn\" (UID: \"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58\") " pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9f223b1-0765-46bb-b8af-136743836d8a-cnibin\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310586    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c290aebc-bdd2-47e7-a307-f00b1df88b5f-serviceca\") pod \"node-ca-klwxw\" (UID: \"c290aebc-bdd2-47e7-a307-f00b1df88b5f\") " pod="openshift-image-registry/node-ca-klwxw"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-run-systemd\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-run-ovn-kubernetes\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.311262 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-run\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-cnibin\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-multus-socket-dir-parent\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310729    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-host\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62gd\" (UniqueName: \"kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd\") pod \"network-check-target-f6494\" (UID: \"62eb9ea7-9552-4c47-a202-adbba7cd45fa\") " pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e441cb9-53b1-41e0-8f86-c0d30c8dd55f-konnectivity-ca\") pod \"konnectivity-agent-42brv\" (UID: \"8e441cb9-53b1-41e0-8f86-c0d30c8dd55f\") " pod="kube-system/konnectivity-agent-42brv"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-etc-kubernetes\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9f223b1-0765-46bb-b8af-136743836d8a-system-cni-dir\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-var-lib-openvswitch\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-sys\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-run-netns\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn9vd\" (UniqueName: \"kubernetes.io/projected/d9f223b1-0765-46bb-b8af-136743836d8a-kube-api-access-gn9vd\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3da572d-acbd-4c3c-a693-c0eac330f719-kubelet-config\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.310994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-socket-dir\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-slash\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.312193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-etc-openvswitch\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-env-overrides\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311054    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-system-cni-dir\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-multus-cni-dir\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-os-release\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a408701-9b8b-48a1-b843-165cb5758437-multus-daemon-config\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-sysconfig\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f4a275b-51e0-4ef7-ba98-196b5589a2d8-host-slash\") pod \"iptables-alerter-5vlnq\" (UID: \"9f4a275b-51e0-4ef7-ba98-196b5589a2d8\") " pod="openshift-network-operator/iptables-alerter-5vlnq"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a408701-9b8b-48a1-b843-165cb5758437-cni-binary-copy\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs\") pod \"network-metrics-daemon-kpcpn\" (UID: \"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58\") " pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d9f223b1-0765-46bb-b8af-136743836d8a-os-release\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-device-dir\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-run-netns\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311439    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-kubelet\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-lib-modules\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.313707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-cni-netd\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-kubernetes\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.311757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-systemd\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.312070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-run-systemd\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.312227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-sysctl-conf\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.312285    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-var-lib-kubelet\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.312333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c290aebc-bdd2-47e7-a307-f00b1df88b5f-host\") pod \"node-ca-klwxw\" (UID: \"c290aebc-bdd2-47e7-a307-f00b1df88b5f\") " pod="openshift-image-registry/node-ca-klwxw"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.312352    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-run-ovn-kubernetes\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.312417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-run\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.312670    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-var-lib-openvswitch\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.312729    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-sys\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.312767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-ovnkube-config\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.312812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-etc-openvswitch\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.312823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-slash\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.312888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-node-log\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.312981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.313030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-sysctl-d\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.313063    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-host-cni-bin\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.314577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.313067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-log-socket\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.315636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.313122    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-systemd-units\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.315636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.313206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f4a275b-51e0-4ef7-ba98-196b5589a2d8-host-slash\") pod \"iptables-alerter-5vlnq\" (UID: \"9f4a275b-51e0-4ef7-ba98-196b5589a2d8\") " pod="openshift-network-operator/iptables-alerter-5vlnq"
Jul 09 18:04:39.315636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.313231    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-env-overrides\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.315636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.313333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-sysconfig\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.315636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.313334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-run-openvswitch\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.315636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.313369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-run-ovn\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.315636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.313381    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-modprobe-d\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.315636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.313609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-ovnkube-script-lib\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.315636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.313661    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f4a275b-51e0-4ef7-ba98-196b5589a2d8-iptables-alerter-script\") pod \"iptables-alerter-5vlnq\" (UID: \"9f4a275b-51e0-4ef7-ba98-196b5589a2d8\") " pod="openshift-network-operator/iptables-alerter-5vlnq"
Jul 09 18:04:39.316144 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.315784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa299791-af91-4d8a-91f2-0b87b0a9732e-tmp\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.316297 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.316270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-ovn-node-metrics-cert\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.316548 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.316524    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aa299791-af91-4d8a-91f2-0b87b0a9732e-etc-tuned\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.335911 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.335864    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:39.335911 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.335914    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:39.336153 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.335930    2578 projected.go:196] Error preparing data for projected volume kube-api-access-j62gd for pod openshift-network-diagnostics/network-check-target-f6494: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:39.336153 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.336020    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd podName:62eb9ea7-9552-4c47-a202-adbba7cd45fa nodeName:}" failed. No retries permitted until 2026-07-09 18:04:39.835995131 +0000 UTC m=+3.022972893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j62gd" (UniqueName: "kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd") pod "network-check-target-f6494" (UID: "62eb9ea7-9552-4c47-a202-adbba7cd45fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:39.336435 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.335861    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2b7t\" (UniqueName: \"kubernetes.io/projected/555b59d8-7ef9-4d32-bfe4-b3ec61808eba-kube-api-access-d2b7t\") pod \"ovnkube-node-nlk4c\" (UID: \"555b59d8-7ef9-4d32-bfe4-b3ec61808eba\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.338207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.338168    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54lqg\" (UniqueName: \"kubernetes.io/projected/aa299791-af91-4d8a-91f2-0b87b0a9732e-kube-api-access-54lqg\") pod \"tuned-qpm9g\" (UID: \"aa299791-af91-4d8a-91f2-0b87b0a9732e\") " pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.339222 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.339193    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw76z\" (UniqueName: \"kubernetes.io/projected/9f4a275b-51e0-4ef7-ba98-196b5589a2d8-kube-api-access-sw76z\") pod \"iptables-alerter-5vlnq\" (UID: \"9f4a275b-51e0-4ef7-ba98-196b5589a2d8\") " pod="openshift-network-operator/iptables-alerter-5vlnq"
Jul 09 18:04:39.340505 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.340475    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj4cx\" (UniqueName: \"kubernetes.io/projected/c290aebc-bdd2-47e7-a307-f00b1df88b5f-kube-api-access-rj4cx\") pod \"node-ca-klwxw\" (UID: \"c290aebc-bdd2-47e7-a307-f00b1df88b5f\") " pod="openshift-image-registry/node-ca-klwxw"
Jul 09 18:04:39.403489 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.403387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal" event={"ID":"fb2f2f9ee8258d42945cd1eb17f2050c","Type":"ContainerStarted","Data":"cf6a313f2856ee669b124b0f1a3162df6be60fe6e9e41ed1f2f74a9b2241e13f"}
Jul 09 18:04:39.405783 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.405746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal" event={"ID":"ecb5e10e9a5b0d3e70008ee6ce282ea9","Type":"ContainerStarted","Data":"43cfb95b34d64a7264595b898c273cfa7851c2f07f51e2b19bf471dffac3cd74"}
Jul 09 18:04:39.412429 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs\") pod \"network-metrics-daemon-kpcpn\" (UID: \"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58\") " pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:39.412429 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d9f223b1-0765-46bb-b8af-136743836d8a-os-release\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.412689 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-device-dir\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.412689 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.412689 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-run-k8s-cni-cncf-io\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.412689 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-multus-conf-dir\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.412689 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r6rv\" (UniqueName: \"kubernetes.io/projected/8a408701-9b8b-48a1-b843-165cb5758437-kube-api-access-2r6rv\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.412689 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-registration-dir\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.412689 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.412592    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:39.412689 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-sys-fs\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.412689 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ccfb877-5ee9-40fa-a28d-8310e8922463-tmp-dir\") pod \"node-resolver-d7pqh\" (UID: \"9ccfb877-5ee9-40fa-a28d-8310e8922463\") " pod="openshift-dns/node-resolver-d7pqh"
Jul 09 18:04:39.412689 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-var-lib-cni-bin\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.412689 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.412686    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs podName:11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:39.912661813 +0000 UTC m=+3.099639568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs") pod "network-metrics-daemon-kpcpn" (UID: "11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:39.413277 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-var-lib-cni-multus\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.413277 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-var-lib-cni-bin\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.413277 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-run-multus-certs\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.413277 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9f223b1-0765-46bb-b8af-136743836d8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.413277 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-device-dir\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.413277 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ccfb877-5ee9-40fa-a28d-8310e8922463-hosts-file\") pod \"node-resolver-d7pqh\" (UID: \"9ccfb877-5ee9-40fa-a28d-8310e8922463\") " pod="openshift-dns/node-resolver-d7pqh"
Jul 09 18:04:39.413277 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-var-lib-kubelet\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.413277 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ccfb877-5ee9-40fa-a28d-8310e8922463-hosts-file\") pod \"node-resolver-d7pqh\" (UID: \"9ccfb877-5ee9-40fa-a28d-8310e8922463\") " pod="openshift-dns/node-resolver-d7pqh"
Jul 09 18:04:39.413277 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.413067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.413277 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.413115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-run-k8s-cni-cncf-io\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.413277 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.413156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-multus-conf-dir\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.418926 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.418198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d9f223b1-0765-46bb-b8af-136743836d8a-os-release\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.418926 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.418317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-var-lib-cni-multus\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.418926 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.418378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-run-multus-certs\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.418926 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.418771    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ccfb877-5ee9-40fa-a28d-8310e8922463-tmp-dir\") pod \"node-resolver-d7pqh\" (UID: \"9ccfb877-5ee9-40fa-a28d-8310e8922463\") " pod="openshift-dns/node-resolver-d7pqh"
Jul 09 18:04:39.418926 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.418839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-registration-dir\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.418926 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.418930    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-sys-fs\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.412951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwc6n\" (UniqueName: \"kubernetes.io/projected/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-kube-api-access-kwc6n\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.418993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-hostroot\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9f223b1-0765-46bb-b8af-136743836d8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e441cb9-53b1-41e0-8f86-c0d30c8dd55f-agent-certs\") pod \"konnectivity-agent-42brv\" (UID: \"8e441cb9-53b1-41e0-8f86-c0d30c8dd55f\") " pod="kube-system/konnectivity-agent-42brv"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9f223b1-0765-46bb-b8af-136743836d8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9f223b1-0765-46bb-b8af-136743836d8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3da572d-acbd-4c3c-a693-c0eac330f719-dbus\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tnd\" (UniqueName: \"kubernetes.io/projected/9ccfb877-5ee9-40fa-a28d-8310e8922463-kube-api-access-z2tnd\") pod \"node-resolver-d7pqh\" (UID: \"9ccfb877-5ee9-40fa-a28d-8310e8922463\") " pod="openshift-dns/node-resolver-d7pqh"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbx4z\" (UniqueName: \"kubernetes.io/projected/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-kube-api-access-jbx4z\") pod \"network-metrics-daemon-kpcpn\" (UID: \"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58\") " pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9f223b1-0765-46bb-b8af-136743836d8a-cnibin\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-cnibin\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-multus-socket-dir-parent\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e441cb9-53b1-41e0-8f86-c0d30c8dd55f-konnectivity-ca\") pod \"konnectivity-agent-42brv\" (UID: \"8e441cb9-53b1-41e0-8f86-c0d30c8dd55f\") " pod="kube-system/konnectivity-agent-42brv"
Jul 09 18:04:39.419327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-etc-kubernetes\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.420061 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9f223b1-0765-46bb-b8af-136743836d8a-system-cni-dir\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.420061 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.420061 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-run-netns\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.420061 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn9vd\" (UniqueName: \"kubernetes.io/projected/d9f223b1-0765-46bb-b8af-136743836d8a-kube-api-access-gn9vd\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.420061 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3da572d-acbd-4c3c-a693-c0eac330f719-kubelet-config\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:39.420061 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-socket-dir\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.420061 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-system-cni-dir\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.420061 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-multus-cni-dir\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.420061 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-os-release\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.420061 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a408701-9b8b-48a1-b843-165cb5758437-multus-daemon-config\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.420061 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.419594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a408701-9b8b-48a1-b843-165cb5758437-cni-binary-copy\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.420061 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.420062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a408701-9b8b-48a1-b843-165cb5758437-cni-binary-copy\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.420671 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.420192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9f223b1-0765-46bb-b8af-136743836d8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.420671 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.420236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-hostroot\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.421441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9f223b1-0765-46bb-b8af-136743836d8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.421471    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-var-lib-kubelet\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.421552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-etc-kubernetes\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.421560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9f223b1-0765-46bb-b8af-136743836d8a-system-cni-dir\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.421641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9f223b1-0765-46bb-b8af-136743836d8a-cnibin\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.421910    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3da572d-acbd-4c3c-a693-c0eac330f719-dbus\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.422119    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9f223b1-0765-46bb-b8af-136743836d8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.422203    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-cnibin\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.422287    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.422356    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret podName:d3da572d-acbd-4c3c-a693-c0eac330f719 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:39.92233433 +0000 UTC m=+3.109312082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret") pod "global-pull-secret-syncer-5kvp7" (UID: "d3da572d-acbd-4c3c-a693-c0eac330f719") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.422448    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-multus-socket-dir-parent\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.422793    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-os-release\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.423129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-socket-dir\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.423216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-host-run-netns\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.423197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e441cb9-53b1-41e0-8f86-c0d30c8dd55f-konnectivity-ca\") pod \"konnectivity-agent-42brv\" (UID: \"8e441cb9-53b1-41e0-8f86-c0d30c8dd55f\") " pod="kube-system/konnectivity-agent-42brv"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.423322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-system-cni-dir\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.424793 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.423328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a408701-9b8b-48a1-b843-165cb5758437-multus-daemon-config\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.425744 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.423510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3da572d-acbd-4c3c-a693-c0eac330f719-kubelet-config\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:39.425744 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.423950    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9f223b1-0765-46bb-b8af-136743836d8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.425744 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.424886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a408701-9b8b-48a1-b843-165cb5758437-multus-cni-dir\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.425744 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.425113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e441cb9-53b1-41e0-8f86-c0d30c8dd55f-agent-certs\") pod \"konnectivity-agent-42brv\" (UID: \"8e441cb9-53b1-41e0-8f86-c0d30c8dd55f\") " pod="kube-system/konnectivity-agent-42brv"
Jul 09 18:04:39.425744 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.425222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.429165 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.429100    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwc6n\" (UniqueName: \"kubernetes.io/projected/e967b6c0-f61b-40cf-9c37-c51b32cb03c2-kube-api-access-kwc6n\") pod \"aws-ebs-csi-driver-node-zxfl9\" (UID: \"e967b6c0-f61b-40cf-9c37-c51b32cb03c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.436960 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.436897    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r6rv\" (UniqueName: \"kubernetes.io/projected/8a408701-9b8b-48a1-b843-165cb5758437-kube-api-access-2r6rv\") pod \"multus-8qxqd\" (UID: \"8a408701-9b8b-48a1-b843-165cb5758437\") " pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.437119 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.437040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbx4z\" (UniqueName: \"kubernetes.io/projected/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-kube-api-access-jbx4z\") pod \"network-metrics-daemon-kpcpn\" (UID: \"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58\") " pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:39.440369 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.440308    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tnd\" (UniqueName: \"kubernetes.io/projected/9ccfb877-5ee9-40fa-a28d-8310e8922463-kube-api-access-z2tnd\") pod \"node-resolver-d7pqh\" (UID: \"9ccfb877-5ee9-40fa-a28d-8310e8922463\") " pod="openshift-dns/node-resolver-d7pqh"
Jul 09 18:04:39.441292 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.441257    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn9vd\" (UniqueName: \"kubernetes.io/projected/d9f223b1-0765-46bb-b8af-136743836d8a-kube-api-access-gn9vd\") pod \"multus-additional-cni-plugins-8jmw8\" (UID: \"d9f223b1-0765-46bb-b8af-136743836d8a\") " pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.506943 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.506869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5vlnq"
Jul 09 18:04:39.516624 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.516373    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:04:39.520694 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:39.520356    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4a275b_51e0_4ef7_ba98_196b5589a2d8.slice/crio-15835abf44bbb66f9e9776f8f44045eb838fd2dfca97f082ff34b003a45b9245 WatchSource:0}: Error finding container 15835abf44bbb66f9e9776f8f44045eb838fd2dfca97f082ff34b003a45b9245: Status 404 returned error can't find the container with id 15835abf44bbb66f9e9776f8f44045eb838fd2dfca97f082ff34b003a45b9245
Jul 09 18:04:39.526899 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.526652    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qpm9g"
Jul 09 18:04:39.527579 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:39.527549    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555b59d8_7ef9_4d32_bfe4_b3ec61808eba.slice/crio-f952a0d11a4a7c19a518e759fc6c548f26e641aa9e77bf5e55f642aad2171059 WatchSource:0}: Error finding container f952a0d11a4a7c19a518e759fc6c548f26e641aa9e77bf5e55f642aad2171059: Status 404 returned error can't find the container with id f952a0d11a4a7c19a518e759fc6c548f26e641aa9e77bf5e55f642aad2171059
Jul 09 18:04:39.534416 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.534221    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-klwxw"
Jul 09 18:04:39.538464 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:39.538425    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa299791_af91_4d8a_91f2_0b87b0a9732e.slice/crio-ebb105dfb88563fe89ea44d3442d799f6f81df07262b1a0afb9f8f3a2d89d2d3 WatchSource:0}: Error finding container ebb105dfb88563fe89ea44d3442d799f6f81df07262b1a0afb9f8f3a2d89d2d3: Status 404 returned error can't find the container with id ebb105dfb88563fe89ea44d3442d799f6f81df07262b1a0afb9f8f3a2d89d2d3
Jul 09 18:04:39.543553 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.543521    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8jmw8"
Jul 09 18:04:39.545967 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:39.545930    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc290aebc_bdd2_47e7_a307_f00b1df88b5f.slice/crio-33b8eb9d411029918e967dd802b952d8370d6da292865f16a8b522394de1542b WatchSource:0}: Error finding container 33b8eb9d411029918e967dd802b952d8370d6da292865f16a8b522394de1542b: Status 404 returned error can't find the container with id 33b8eb9d411029918e967dd802b952d8370d6da292865f16a8b522394de1542b
Jul 09 18:04:39.551905 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.551632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d7pqh"
Jul 09 18:04:39.560093 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:39.560040    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f223b1_0765_46bb_b8af_136743836d8a.slice/crio-31eaedd0ee9bbafdcc13ee46897696c776cc81f66ecede2c08a373ca529ba0e8 WatchSource:0}: Error finding container 31eaedd0ee9bbafdcc13ee46897696c776cc81f66ecede2c08a373ca529ba0e8: Status 404 returned error can't find the container with id 31eaedd0ee9bbafdcc13ee46897696c776cc81f66ecede2c08a373ca529ba0e8
Jul 09 18:04:39.561656 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.561629    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-42brv"
Jul 09 18:04:39.565400 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:39.565365    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccfb877_5ee9_40fa_a28d_8310e8922463.slice/crio-4ef915d05e15084f8395ad6288d69f9be69e1f37302fbdb5d1ad822f48ad7fe3 WatchSource:0}: Error finding container 4ef915d05e15084f8395ad6288d69f9be69e1f37302fbdb5d1ad822f48ad7fe3: Status 404 returned error can't find the container with id 4ef915d05e15084f8395ad6288d69f9be69e1f37302fbdb5d1ad822f48ad7fe3
Jul 09 18:04:39.571737 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.571704    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9"
Jul 09 18:04:39.572036 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:39.571981    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e441cb9_53b1_41e0_8f86_c0d30c8dd55f.slice/crio-2d3454523ab6fa6b7992b0d6c159e28db03301fb3e45afed21d768a169adb3a9 WatchSource:0}: Error finding container 2d3454523ab6fa6b7992b0d6c159e28db03301fb3e45afed21d768a169adb3a9: Status 404 returned error can't find the container with id 2d3454523ab6fa6b7992b0d6c159e28db03301fb3e45afed21d768a169adb3a9
Jul 09 18:04:39.581012 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.580860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8qxqd"
Jul 09 18:04:39.582771 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:39.582735    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode967b6c0_f61b_40cf_9c37_c51b32cb03c2.slice/crio-561f87ccfb9eadf649f4cfa10244f94088ed949440aab611a4b6d0ba4eda9e91 WatchSource:0}: Error finding container 561f87ccfb9eadf649f4cfa10244f94088ed949440aab611a4b6d0ba4eda9e91: Status 404 returned error can't find the container with id 561f87ccfb9eadf649f4cfa10244f94088ed949440aab611a4b6d0ba4eda9e91
Jul 09 18:04:39.590681 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:04:39.590651    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a408701_9b8b_48a1_b843_165cb5758437.slice/crio-d95152c8910f3d5ce8ac88175bd8d8ccc644abae37fd45c2f782007ffaa64ead WatchSource:0}: Error finding container d95152c8910f3d5ce8ac88175bd8d8ccc644abae37fd45c2f782007ffaa64ead: Status 404 returned error can't find the container with id d95152c8910f3d5ce8ac88175bd8d8ccc644abae37fd45c2f782007ffaa64ead
Jul 09 18:04:39.609657 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.609605    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:39.924878 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.924784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:39.924878 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.924845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62gd\" (UniqueName: \"kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd\") pod \"network-check-target-f6494\" (UID: \"62eb9ea7-9552-4c47-a202-adbba7cd45fa\") " pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:39.925082 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:39.924896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs\") pod \"network-metrics-daemon-kpcpn\" (UID: \"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58\") " pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:39.925082 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.925007    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:39.925082 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.925023    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:39.925082 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.925050    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:39.925082 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.925063    2578 projected.go:196] Error preparing data for projected volume kube-api-access-j62gd for pod openshift-network-diagnostics/network-check-target-f6494: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:39.925082 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.925079    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs podName:11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:40.925056623 +0000 UTC m=+4.112034370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs") pod "network-metrics-daemon-kpcpn" (UID: "11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:39.925339 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.925009    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:39.925339 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.925116    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd podName:62eb9ea7-9552-4c47-a202-adbba7cd45fa nodeName:}" failed. No retries permitted until 2026-07-09 18:04:40.925096441 +0000 UTC m=+4.112074211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j62gd" (UniqueName: "kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd") pod "network-check-target-f6494" (UID: "62eb9ea7-9552-4c47-a202-adbba7cd45fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:39.925339 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:39.925139    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret podName:d3da572d-acbd-4c3c-a693-c0eac330f719 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:40.925125305 +0000 UTC m=+4.112103056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret") pod "global-pull-secret-syncer-5kvp7" (UID: "d3da572d-acbd-4c3c-a693-c0eac330f719") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:40.294731 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.294575    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:39 +0000 UTC" deadline="2028-04-19 15:33:05.842835722 +0000 UTC"
Jul 09 18:04:40.294731 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.294622    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15597h28m25.548219196s"
Jul 09 18:04:40.398001 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.397938    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:40.398201 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:40.398139    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:04:40.409696 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.409637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8qxqd" event={"ID":"8a408701-9b8b-48a1-b843-165cb5758437","Type":"ContainerStarted","Data":"d95152c8910f3d5ce8ac88175bd8d8ccc644abae37fd45c2f782007ffaa64ead"}
Jul 09 18:04:40.411174 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.411136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9" event={"ID":"e967b6c0-f61b-40cf-9c37-c51b32cb03c2","Type":"ContainerStarted","Data":"561f87ccfb9eadf649f4cfa10244f94088ed949440aab611a4b6d0ba4eda9e91"}
Jul 09 18:04:40.412792 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.412761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jmw8" event={"ID":"d9f223b1-0765-46bb-b8af-136743836d8a","Type":"ContainerStarted","Data":"31eaedd0ee9bbafdcc13ee46897696c776cc81f66ecede2c08a373ca529ba0e8"}
Jul 09 18:04:40.414776 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.414649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-klwxw" event={"ID":"c290aebc-bdd2-47e7-a307-f00b1df88b5f","Type":"ContainerStarted","Data":"33b8eb9d411029918e967dd802b952d8370d6da292865f16a8b522394de1542b"}
Jul 09 18:04:40.416990 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.416953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c" event={"ID":"555b59d8-7ef9-4d32-bfe4-b3ec61808eba","Type":"ContainerStarted","Data":"f952a0d11a4a7c19a518e759fc6c548f26e641aa9e77bf5e55f642aad2171059"}
Jul 09 18:04:40.418600 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.418570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5vlnq" event={"ID":"9f4a275b-51e0-4ef7-ba98-196b5589a2d8","Type":"ContainerStarted","Data":"15835abf44bbb66f9e9776f8f44045eb838fd2dfca97f082ff34b003a45b9245"}
Jul 09 18:04:40.421098 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.421054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-42brv" event={"ID":"8e441cb9-53b1-41e0-8f86-c0d30c8dd55f","Type":"ContainerStarted","Data":"2d3454523ab6fa6b7992b0d6c159e28db03301fb3e45afed21d768a169adb3a9"}
Jul 09 18:04:40.422902 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.422819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d7pqh" event={"ID":"9ccfb877-5ee9-40fa-a28d-8310e8922463","Type":"ContainerStarted","Data":"4ef915d05e15084f8395ad6288d69f9be69e1f37302fbdb5d1ad822f48ad7fe3"}
Jul 09 18:04:40.425130 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.425100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qpm9g" event={"ID":"aa299791-af91-4d8a-91f2-0b87b0a9732e","Type":"ContainerStarted","Data":"ebb105dfb88563fe89ea44d3442d799f6f81df07262b1a0afb9f8f3a2d89d2d3"}
Jul 09 18:04:40.936058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.935153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:40.936058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.935427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62gd\" (UniqueName: \"kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd\") pod \"network-check-target-f6494\" (UID: \"62eb9ea7-9552-4c47-a202-adbba7cd45fa\") " pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:40.936058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:40.935473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs\") pod \"network-metrics-daemon-kpcpn\" (UID: \"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58\") " pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:40.936058 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:40.935611    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:40.936058 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:40.935674    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs podName:11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:42.9356548 +0000 UTC m=+6.122632549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs") pod "network-metrics-daemon-kpcpn" (UID: "11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:40.941910 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:40.936787    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:40.941910 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:40.936892    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret podName:d3da572d-acbd-4c3c-a693-c0eac330f719 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:42.936862967 +0000 UTC m=+6.123840718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret") pod "global-pull-secret-syncer-5kvp7" (UID: "d3da572d-acbd-4c3c-a693-c0eac330f719") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:40.941910 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:40.938415    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:40.941910 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:40.938440    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:40.941910 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:40.938461    2578 projected.go:196] Error preparing data for projected volume kube-api-access-j62gd for pod openshift-network-diagnostics/network-check-target-f6494: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:40.941910 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:40.938519    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd podName:62eb9ea7-9552-4c47-a202-adbba7cd45fa nodeName:}" failed. No retries permitted until 2026-07-09 18:04:42.938495309 +0000 UTC m=+6.125473070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j62gd" (UniqueName: "kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd") pod "network-check-target-f6494" (UID: "62eb9ea7-9552-4c47-a202-adbba7cd45fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:41.297049 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:41.296928    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:39 +0000 UTC" deadline="2028-01-28 06:30:09.612214454 +0000 UTC"
Jul 09 18:04:41.297049 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:41.296990    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13620h25m28.315229014s"
Jul 09 18:04:41.399998 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:41.399971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:41.400179 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:41.400103    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:04:41.400528 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:41.400508    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:41.400674 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:41.400649    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:04:42.398178 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:42.398139    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:42.398691 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:42.398276    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:04:42.951522 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:42.951480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:42.951696 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:42.951533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62gd\" (UniqueName: \"kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd\") pod \"network-check-target-f6494\" (UID: \"62eb9ea7-9552-4c47-a202-adbba7cd45fa\") " pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:42.951696 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:42.951572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs\") pod \"network-metrics-daemon-kpcpn\" (UID: \"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58\") " pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:42.951910 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:42.951726    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:42.951910 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:42.951788    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs podName:11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:46.951770052 +0000 UTC m=+10.138747802 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs") pod "network-metrics-daemon-kpcpn" (UID: "11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:42.952259 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:42.952219    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:42.952372 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:42.952284    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret podName:d3da572d-acbd-4c3c-a693-c0eac330f719 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:46.952267404 +0000 UTC m=+10.139245151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret") pod "global-pull-secret-syncer-5kvp7" (UID: "d3da572d-acbd-4c3c-a693-c0eac330f719") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:42.952372 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:42.952364    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:42.952476 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:42.952379    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:42.952476 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:42.952391    2578 projected.go:196] Error preparing data for projected volume kube-api-access-j62gd for pod openshift-network-diagnostics/network-check-target-f6494: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:42.952476 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:42.952427    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd podName:62eb9ea7-9552-4c47-a202-adbba7cd45fa nodeName:}" failed. No retries permitted until 2026-07-09 18:04:46.952414785 +0000 UTC m=+10.139392535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j62gd" (UniqueName: "kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd") pod "network-check-target-f6494" (UID: "62eb9ea7-9552-4c47-a202-adbba7cd45fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:43.399541 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:43.399460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:43.399541 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:43.399486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:43.399984 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:43.399608    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:04:43.399984 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:43.399740    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:04:44.398357 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:44.397811    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:44.398357 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:44.397957    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:04:45.398104 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:45.397878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:45.398104 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:45.398042    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:04:45.398626 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:45.398390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:45.398626 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:45.398503    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:04:46.398248 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:46.398209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:46.398737 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:46.398354    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:04:46.989805 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:46.989749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:46.989981 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:46.989824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62gd\" (UniqueName: \"kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd\") pod \"network-check-target-f6494\" (UID: \"62eb9ea7-9552-4c47-a202-adbba7cd45fa\") " pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:46.989981 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:46.989866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs\") pod \"network-metrics-daemon-kpcpn\" (UID: \"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58\") " pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:46.989981 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:46.989899    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:46.989981 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:46.989968    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret podName:d3da572d-acbd-4c3c-a693-c0eac330f719 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:54.989948083 +0000 UTC m=+18.176925847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret") pod "global-pull-secret-syncer-5kvp7" (UID: "d3da572d-acbd-4c3c-a693-c0eac330f719") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:46.989981 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:46.989979    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:46.990258 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:46.990013    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs podName:11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:54.99000268 +0000 UTC m=+18.176980431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs") pod "network-metrics-daemon-kpcpn" (UID: "11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:46.990258 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:46.990082    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:46.990258 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:46.990094    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:46.990258 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:46.990106    2578 projected.go:196] Error preparing data for projected volume kube-api-access-j62gd for pod openshift-network-diagnostics/network-check-target-f6494: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:46.990258 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:46.990137    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd podName:62eb9ea7-9552-4c47-a202-adbba7cd45fa nodeName:}" failed. No retries permitted until 2026-07-09 18:04:54.990127633 +0000 UTC m=+18.177105379 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j62gd" (UniqueName: "kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd") pod "network-check-target-f6494" (UID: "62eb9ea7-9552-4c47-a202-adbba7cd45fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:47.398317 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:47.398197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:47.398779 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:47.398316    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:04:47.398779 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:47.398714    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:47.398931 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:47.398837    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:04:48.397274 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:48.397187    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:48.397443 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:48.397335    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:04:49.397588 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:49.397544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:49.398054 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:49.397685    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:04:49.398054 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:49.397747    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:49.398054 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:49.397899    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:04:50.397999 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:50.397955    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:50.398477 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:50.398106    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:04:51.397430 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:51.397394    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:51.397610 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:51.397394    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:51.397610 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:51.397554    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:04:51.397717 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:51.397628    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:04:52.397764 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:52.397721    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:52.398294 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:52.397867    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:04:53.397744 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:53.397699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:53.397959 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:53.397867    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:04:53.397959 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:53.397700    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:53.398363 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:53.398033    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:04:54.398136 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:54.398095    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:54.398568 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:54.398242    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:04:55.051060 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:55.051021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:55.051060 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:55.051061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62gd\" (UniqueName: \"kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd\") pod \"network-check-target-f6494\" (UID: \"62eb9ea7-9552-4c47-a202-adbba7cd45fa\") " pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:55.051341 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:55.051087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs\") pod \"network-metrics-daemon-kpcpn\" (UID: \"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58\") " pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:55.051341 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:55.051191    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:55.051341 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:55.051247    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:55.051341 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:55.051283    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret podName:d3da572d-acbd-4c3c-a693-c0eac330f719 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:11.051261442 +0000 UTC m=+34.238239192 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret") pod "global-pull-secret-syncer-5kvp7" (UID: "d3da572d-acbd-4c3c-a693-c0eac330f719") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:55.051341 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:55.051250    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:55.051341 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:55.051305    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs podName:11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:11.051294529 +0000 UTC m=+34.238272281 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs") pod "network-metrics-daemon-kpcpn" (UID: "11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:55.051341 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:55.051313    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:55.051341 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:55.051328    2578 projected.go:196] Error preparing data for projected volume kube-api-access-j62gd for pod openshift-network-diagnostics/network-check-target-f6494: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:55.051689 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:55.051374    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd podName:62eb9ea7-9552-4c47-a202-adbba7cd45fa nodeName:}" failed. No retries permitted until 2026-07-09 18:05:11.051359367 +0000 UTC m=+34.238337117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j62gd" (UniqueName: "kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd") pod "network-check-target-f6494" (UID: "62eb9ea7-9552-4c47-a202-adbba7cd45fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:55.398368 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:55.398274    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:55.398826 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:55.398274    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:55.398826 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:55.398426    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:04:55.398826 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:55.398495    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:04:56.397629 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:56.397589    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:56.397848 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:56.397714    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:04:57.398361 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:57.398320    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:57.398755 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:57.398464    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:04:57.398755 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:57.398505    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:57.398755 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:57.398638    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:04:58.397696 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:58.397662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:04:58.397912 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:58.397784    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:04:59.398101 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.397753    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:04:59.399073 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.397750    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:04:59.399073 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:59.398196    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:04:59.399073 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:04:59.398280    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:04:59.474349 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.474313    2578 generic.go:356] "Generic (PLEG): container finished" podID="d9f223b1-0765-46bb-b8af-136743836d8a" containerID="74ccb27bafca7969b42c5c20c107996cbb52041ee76da19f2e1c0a880926bd89" exitCode=0
Jul 09 18:04:59.474546 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.474406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jmw8" event={"ID":"d9f223b1-0765-46bb-b8af-136743836d8a","Type":"ContainerDied","Data":"74ccb27bafca7969b42c5c20c107996cbb52041ee76da19f2e1c0a880926bd89"}
Jul 09 18:04:59.476079 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.476042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-klwxw" event={"ID":"c290aebc-bdd2-47e7-a307-f00b1df88b5f","Type":"ContainerStarted","Data":"478b7a39c33b19996a9fb186ab0c0efdfb16febab67cb986c087850db5ded091"}
Jul 09 18:04:59.478705 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.478683    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:04:59.479110 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.479083    2578 generic.go:356] "Generic (PLEG): container finished" podID="555b59d8-7ef9-4d32-bfe4-b3ec61808eba" containerID="8bfc49e6c0029501caf8809a718dfc569be2034a910c6a0d35e12c975bcdce96" exitCode=1
Jul 09 18:04:59.479187 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.479127    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c" event={"ID":"555b59d8-7ef9-4d32-bfe4-b3ec61808eba","Type":"ContainerStarted","Data":"f739e1e867ab55a8c10195013b5370b9aa562e10ba07760e5b2ba8b07782e4bf"}
Jul 09 18:04:59.479232 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.479217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c" event={"ID":"555b59d8-7ef9-4d32-bfe4-b3ec61808eba","Type":"ContainerStarted","Data":"8f85e262c21fbdd9dd5770b202e7cc1a08c47130b2a279ca5610f85ff781c8b3"}
Jul 09 18:04:59.479300 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.479269    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c" event={"ID":"555b59d8-7ef9-4d32-bfe4-b3ec61808eba","Type":"ContainerStarted","Data":"f96b619894bc662d4d958d3618ec74ab4ff74936c4b887773f2b1dbe3842bb18"}
Jul 09 18:04:59.479358 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.479312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c" event={"ID":"555b59d8-7ef9-4d32-bfe4-b3ec61808eba","Type":"ContainerDied","Data":"8bfc49e6c0029501caf8809a718dfc569be2034a910c6a0d35e12c975bcdce96"}
Jul 09 18:04:59.479394 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.479363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c" event={"ID":"555b59d8-7ef9-4d32-bfe4-b3ec61808eba","Type":"ContainerStarted","Data":"d5c539eaef14f4a66a291ed1959919a546fff810b8048608dbc8291dda123f4d"}
Jul 09 18:04:59.484268 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.484237    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal" event={"ID":"fb2f2f9ee8258d42945cd1eb17f2050c","Type":"ContainerStarted","Data":"58d472eb198bc0c96ed08f686dfea94249cf1a65a4ffd89ce4f5376731eea312"}
Jul 09 18:04:59.485667 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.485643    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-42brv" event={"ID":"8e441cb9-53b1-41e0-8f86-c0d30c8dd55f","Type":"ContainerStarted","Data":"b78a12317d8606d66fff1d84a162d4691c3226903dfaa6a572bc8b48292309fd"}
Jul 09 18:04:59.487229 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.487207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d7pqh" event={"ID":"9ccfb877-5ee9-40fa-a28d-8310e8922463","Type":"ContainerStarted","Data":"d58086829ecf81c11a9149447df264fe14c6142c2a80e227bc0933dbf5b1e3d2"}
Jul 09 18:04:59.488605 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.488579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qpm9g" event={"ID":"aa299791-af91-4d8a-91f2-0b87b0a9732e","Type":"ContainerStarted","Data":"1f8c37601a02aeac5309a549cf6f018128d347a7828899aab658a6432eb51ce8"}
Jul 09 18:04:59.489880 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.489852    2578 generic.go:356] "Generic (PLEG): container finished" podID="ecb5e10e9a5b0d3e70008ee6ce282ea9" containerID="2a8f13dc12eba6437108fc4dd2901c9bc69c332076c129ab9278a062f41c8bb0" exitCode=0
Jul 09 18:04:59.489984 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.489922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal" event={"ID":"ecb5e10e9a5b0d3e70008ee6ce282ea9","Type":"ContainerDied","Data":"2a8f13dc12eba6437108fc4dd2901c9bc69c332076c129ab9278a062f41c8bb0"}
Jul 09 18:04:59.491276 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.491256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8qxqd" event={"ID":"8a408701-9b8b-48a1-b843-165cb5758437","Type":"ContainerStarted","Data":"0f99bdd0e11cd9d01a77549f40e5904b3066d01860933c4c54dcffcec0ac67fe"}
Jul 09 18:04:59.492665 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.492646    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9" event={"ID":"e967b6c0-f61b-40cf-9c37-c51b32cb03c2","Type":"ContainerStarted","Data":"dfc8159c2661c1dda3b02083b14909090bb8ab02587c4f5d0a2471c61e0cb12d"}
Jul 09 18:04:59.525782 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.525739    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-42brv" podStartSLOduration=11.426221839 podStartE2EDuration="22.525722344s" totalImagesPullingTime="11.099500505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:59.525437828 +0000 UTC m=+22.712415595" watchObservedRunningTime="2026-07-09 18:04:59.525722344 +0000 UTC m=+22.712700111"
Jul 09 18:04:59.597269 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.597216    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8qxqd" podStartSLOduration=3.261592748 podStartE2EDuration="22.597198515s" totalImagesPullingTime="19.335605767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:59.558266176 +0000 UTC m=+22.745243982" watchObservedRunningTime="2026-07-09 18:04:59.597198515 +0000 UTC m=+22.784176283"
Jul 09 18:04:59.652653 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.652594    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal" podStartSLOduration=21.652578698 podStartE2EDuration="21.652578698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:59.623879196 +0000 UTC m=+22.810856987" watchObservedRunningTime="2026-07-09 18:04:59.652578698 +0000 UTC m=+22.839556474"
Jul 09 18:04:59.652867 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.652664    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qpm9g" podStartSLOduration=3.314102621 podStartE2EDuration="22.652659862s" totalImagesPullingTime="19.338557241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:59.652458412 +0000 UTC m=+22.839436178" watchObservedRunningTime="2026-07-09 18:04:59.652659862 +0000 UTC m=+22.839637630"
Jul 09 18:04:59.672026 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:04:59.671946    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-klwxw" podStartSLOduration=3.350067318 podStartE2EDuration="22.671928269s" totalImagesPullingTime="19.321860951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:59.671369583 +0000 UTC m=+22.858347378" watchObservedRunningTime="2026-07-09 18:04:59.671928269 +0000 UTC m=+22.858906036"
Jul 09 18:05:00.347232 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:00.347192    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-42brv"
Jul 09 18:05:00.347843 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:00.347822    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-42brv"
Jul 09 18:05:00.369683 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:00.369630    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d7pqh" podStartSLOduration=12.263796008 podStartE2EDuration="23.369611224s" totalImagesPullingTime="11.105815216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:59.696757029 +0000 UTC m=+22.883734797" watchObservedRunningTime="2026-07-09 18:05:00.369611224 +0000 UTC m=+23.556589003"
Jul 09 18:05:00.397663 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:00.397625    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:05:00.397871 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:00.397750    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:05:00.498761 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:00.498736    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:05:00.499275 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:00.499245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c" event={"ID":"555b59d8-7ef9-4d32-bfe4-b3ec61808eba","Type":"ContainerStarted","Data":"820ebe49e932108eb29a24827e29801ee50cda5603d499b267577d9245786524"}
Jul 09 18:05:00.500862 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:00.500830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5vlnq" event={"ID":"9f4a275b-51e0-4ef7-ba98-196b5589a2d8","Type":"ContainerStarted","Data":"2950d2da53ada0454a84ae69b4820fd3ec5197136a3dca9bbac94e2c5f9a2780"}
Jul 09 18:05:00.502981 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:00.502950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal" event={"ID":"ecb5e10e9a5b0d3e70008ee6ce282ea9","Type":"ContainerStarted","Data":"90a57ee410bac2ed1fb4d947f061bcf066baa1d42a2d36d9b62a2d49d52933ed"}
Jul 09 18:05:00.519649 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:00.519590    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5vlnq" podStartSLOduration=12.368424518 podStartE2EDuration="23.519570163s" totalImagesPullingTime="11.151145645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:00.519480146 +0000 UTC m=+23.706457916" watchObservedRunningTime="2026-07-09 18:05:00.519570163 +0000 UTC m=+23.706547942"
Jul 09 18:05:00.537020 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:00.536959    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-79.ec2.internal" podStartSLOduration=22.536939079 podStartE2EDuration="22.536939079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:00.535990096 +0000 UTC m=+23.722967864" watchObservedRunningTime="2026-07-09 18:05:00.536939079 +0000 UTC m=+23.723916848"
Jul 09 18:05:00.581501 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:00.581423    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 18:05:01.271112 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:01.270952    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:05:00.581446831Z","UUID":"e5c4e257-53cc-4b04-ac89-bf71f372d024","Handler":null,"Name":"","Endpoint":""}
Jul 09 18:05:01.273204 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:01.273171    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 18:05:01.273346 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:01.273219    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 18:05:01.397517 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:01.397480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:05:01.397700 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:01.397480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:05:01.397700 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:01.397633    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:05:01.397700 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:01.397681    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:05:01.507622 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:01.507585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9" event={"ID":"e967b6c0-f61b-40cf-9c37-c51b32cb03c2","Type":"ContainerStarted","Data":"f080a84ea1e9e544decbd443ae9ffe7396aaa331c732b59609ea5d6760813066"}
Jul 09 18:05:01.508072 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:01.507655    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 09 18:05:02.397838 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:02.397635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:05:02.398075 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:02.397918    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:05:02.512891 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:02.512852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9" event={"ID":"e967b6c0-f61b-40cf-9c37-c51b32cb03c2","Type":"ContainerStarted","Data":"7758cd84dac1e937003939fe730a6fc8b713eec582916100780173b13e24e177"}
Jul 09 18:05:02.516304 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:02.516279    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:05:02.516715 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:02.516689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c" event={"ID":"555b59d8-7ef9-4d32-bfe4-b3ec61808eba","Type":"ContainerStarted","Data":"605867d1494352d41ccb9aa079020d57bbf69431bd13c12cbff81bb325d31dc5"}
Jul 09 18:05:02.540533 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:02.540480    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxfl9" podStartSLOduration=4.007112466 podStartE2EDuration="25.540462369s" totalImagesPullingTime="21.533349903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:02.540368765 +0000 UTC m=+25.727346534" watchObservedRunningTime="2026-07-09 18:05:02.540462369 +0000 UTC m=+25.727440136"
Jul 09 18:05:03.071632 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:03.071591    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-42brv"
Jul 09 18:05:03.071886 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:03.071729    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 09 18:05:03.072439 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:03.072415    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-42brv"
Jul 09 18:05:03.397703 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:03.397599    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:05:03.397703 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:03.397631    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:05:03.397950 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:03.397770    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:05:03.398164 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:03.398119    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:05:04.398229 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:04.398042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:05:04.398962 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:04.398300    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:05:04.522536 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:04.522496    2578 generic.go:356] "Generic (PLEG): container finished" podID="d9f223b1-0765-46bb-b8af-136743836d8a" containerID="99d5f500d1b152e301eb86109df529616412aace8f0d95e73b15cef4d60aac55" exitCode=0
Jul 09 18:05:04.522740 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:04.522591    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jmw8" event={"ID":"d9f223b1-0765-46bb-b8af-136743836d8a","Type":"ContainerDied","Data":"99d5f500d1b152e301eb86109df529616412aace8f0d95e73b15cef4d60aac55"}
Jul 09 18:05:04.526145 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:04.526128    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:05:04.526483 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:04.526457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c" event={"ID":"555b59d8-7ef9-4d32-bfe4-b3ec61808eba","Type":"ContainerStarted","Data":"42da1671b54de1f59cf067d7101c5c0367e7b87837f4bee796d213b6b964af22"}
Jul 09 18:05:04.526773 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:04.526749    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:05:04.526844 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:04.526832    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:05:04.526892 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:04.526884    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:05:04.526970 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:04.526958    2578 scope.go:117] "RemoveContainer" containerID="8bfc49e6c0029501caf8809a718dfc569be2034a910c6a0d35e12c975bcdce96"
Jul 09 18:05:04.544756 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:04.544731    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:05:04.545041 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:04.545025    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:05:05.398099 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:05.398064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:05:05.398334 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:05.398067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:05:05.398334 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:05.398202    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:05:05.398692 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:05.398334    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:05:05.531405 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:05.531373    2578 generic.go:356] "Generic (PLEG): container finished" podID="d9f223b1-0765-46bb-b8af-136743836d8a" containerID="7a436e2a936751112c305a508d0dc40fb67ed228807d0557ad5b4736d4d9f51b" exitCode=0
Jul 09 18:05:05.531583 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:05.531452    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jmw8" event={"ID":"d9f223b1-0765-46bb-b8af-136743836d8a","Type":"ContainerDied","Data":"7a436e2a936751112c305a508d0dc40fb67ed228807d0557ad5b4736d4d9f51b"}
Jul 09 18:05:05.535402 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:05.535380    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:05:05.535750 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:05.535725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c" event={"ID":"555b59d8-7ef9-4d32-bfe4-b3ec61808eba","Type":"ContainerStarted","Data":"6d478e09152768c4523e443a85407fcf509cd590d7dcce23b5868490e83efbbc"}
Jul 09 18:05:06.397871 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:06.397835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:05:06.398044 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:06.397951    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:05:06.521984 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:06.521919    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c" podStartSLOduration=10.107325185 podStartE2EDuration="29.521892259s" totalImagesPullingTime="19.414567074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:05.626240856 +0000 UTC m=+28.813218624" watchObservedRunningTime="2026-07-09 18:05:06.521892259 +0000 UTC m=+29.708870027"
Jul 09 18:05:06.522455 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:06.522259    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5kvp7"]
Jul 09 18:05:06.522455 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:06.522379    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:05:06.522569 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:06.522500    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:05:06.523718 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:06.523693    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f6494"]
Jul 09 18:05:06.529054 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:06.529026    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kpcpn"]
Jul 09 18:05:06.529219 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:06.529144    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:05:06.529302 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:06.529259    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:05:06.539603 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:06.539579    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:05:06.539766 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:06.539685    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:05:07.544092 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:07.544057    2578 generic.go:356] "Generic (PLEG): container finished" podID="d9f223b1-0765-46bb-b8af-136743836d8a" containerID="ada97d6ccb3c6e2fd89ca0209f229331ea0ca3641b96e278061ab429fd94abd7" exitCode=0
Jul 09 18:05:07.544442 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:07.544118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jmw8" event={"ID":"d9f223b1-0765-46bb-b8af-136743836d8a","Type":"ContainerDied","Data":"ada97d6ccb3c6e2fd89ca0209f229331ea0ca3641b96e278061ab429fd94abd7"}
Jul 09 18:05:08.397270 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:08.397184    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:05:08.397423 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:08.397331    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:05:08.397492 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:08.397430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:05:08.397651 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:08.397611    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:05:08.397739 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:08.397665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:05:08.397805 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:08.397749    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:05:10.397871 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:10.397807    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:05:10.398373 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:10.397967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:05:10.398373 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:10.397972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:05:10.398373 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:10.398048    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5kvp7" podUID="d3da572d-acbd-4c3c-a693-c0eac330f719"
Jul 09 18:05:10.398373 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:10.398117    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6494" podUID="62eb9ea7-9552-4c47-a202-adbba7cd45fa"
Jul 09 18:05:10.398373 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:10.398189    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpcpn" podUID="11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58"
Jul 09 18:05:11.075818 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.075761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:05:11.076015 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.075834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62gd\" (UniqueName: \"kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd\") pod \"network-check-target-f6494\" (UID: \"62eb9ea7-9552-4c47-a202-adbba7cd45fa\") " pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:05:11.076015 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.075874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs\") pod \"network-metrics-daemon-kpcpn\" (UID: \"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58\") " pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:05:11.076015 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.075931    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:05:11.076015 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.075975    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:05:11.076015 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.075977    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:05:11.076015 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.076012    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret podName:d3da572d-acbd-4c3c-a693-c0eac330f719 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:43.075991287 +0000 UTC m=+66.262969050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret") pod "global-pull-secret-syncer-5kvp7" (UID: "d3da572d-acbd-4c3c-a693-c0eac330f719") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:05:11.076244 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.076030    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:05:11.076244 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.076043    2578 projected.go:196] Error preparing data for projected volume kube-api-access-j62gd for pod openshift-network-diagnostics/network-check-target-f6494: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:05:11.076244 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.076033    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs podName:11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:43.076022908 +0000 UTC m=+66.263000655 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs") pod "network-metrics-daemon-kpcpn" (UID: "11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:05:11.076244 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.076100    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd podName:62eb9ea7-9552-4c47-a202-adbba7cd45fa nodeName:}" failed. No retries permitted until 2026-07-09 18:05:43.07608093 +0000 UTC m=+66.263058682 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j62gd" (UniqueName: "kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd") pod "network-check-target-f6494" (UID: "62eb9ea7-9552-4c47-a202-adbba7cd45fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:05:11.681636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.681545    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-79.ec2.internal" event="NodeReady"
Jul 09 18:05:11.682102 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.681700    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 18:05:11.738808 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.738759    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz"]
Jul 09 18:05:11.743627 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.743597    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-675d564c74-s5fsx"]
Jul 09 18:05:11.743785 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.743736    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz"
Jul 09 18:05:11.747076 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.747048    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:11.749119 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.749095    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"]
Jul 09 18:05:11.752281 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.752253    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 09 18:05:11.752417 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.752385    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6846c96d87-7mhdw"]
Jul 09 18:05:11.752541 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.752525    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:11.753296 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.753046    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 09 18:05:11.753870 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.753639    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 09 18:05:11.753870 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.753677    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 18:05:11.753997 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.753953    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:05:11.753997 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.753961    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7lpn7\""
Jul 09 18:05:11.754150 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.754131    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 09 18:05:11.754228 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.754172    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 09 18:05:11.754419 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.754399    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rfwbw\""
Jul 09 18:05:11.755205 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.755185    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 18:05:11.755283 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.755243    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 09 18:05:11.755283 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.755262    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 18:05:11.755650 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.755633    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 09 18:05:11.756152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.756132    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-kln87"]
Jul 09 18:05:11.756327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.756309    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.757487 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.757466    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 18:05:11.757566 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.757545    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 09 18:05:11.757616 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.757605    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nt6vh\""
Jul 09 18:05:11.758825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.758790    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 18:05:11.759434 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.759419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:11.759927 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.759579    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 18:05:11.759927 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.759853    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4mr58\""
Jul 09 18:05:11.759927 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.759882    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 09 18:05:11.765320 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.764951    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 18:05:11.765320 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.764959    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:05:11.765494 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.765411    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 09 18:05:11.766782 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.766760    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zwv4b\""
Jul 09 18:05:11.768365 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.767999    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 09 18:05:11.768365 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.768189    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 09 18:05:11.771250 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.771226    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz"]
Jul 09 18:05:11.776291 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.776263    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 18:05:11.776869 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.776837    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw"]
Jul 09 18:05:11.778644 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.778624    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 09 18:05:11.780223 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.780201    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-675d564c74-s5fsx"]
Jul 09 18:05:11.780325 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.780306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw"
Jul 09 18:05:11.783032 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.783010    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:05:11.786556 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.783984    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 09 18:05:11.786556 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.784549    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 09 18:05:11.786556 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.784717    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"]
Jul 09 18:05:11.786556 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.785093    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-n2zsj\""
Jul 09 18:05:11.786556 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.785381    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 09 18:05:11.786890 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.786665    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-kln87"]
Jul 09 18:05:11.786890 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.786746    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pf6x5"]
Jul 09 18:05:11.793719 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.793695    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6846c96d87-7mhdw"]
Jul 09 18:05:11.793859 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.793740    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-ccd5k"]
Jul 09 18:05:11.793931 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.793911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pf6x5"
Jul 09 18:05:11.796790 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.796769    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 09 18:05:11.797104 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.797082    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:05:11.797569 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.797549    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-t2m79\""
Jul 09 18:05:11.797669 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.797612    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:11.807195 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.807169    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 18:05:11.807388 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.807368    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 09 18:05:11.807483 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.807471    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5q9nr\""
Jul 09 18:05:11.807612 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.807593    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 09 18:05:11.807777 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.807752    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 18:05:11.816185 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.816159    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 09 18:05:11.819839 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.819787    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw"]
Jul 09 18:05:11.820482 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.820462    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p7zkv"]
Jul 09 18:05:11.823859 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.823837    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"]
Jul 09 18:05:11.825015 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.824210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:11.827552 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.827533    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nq7m5\""
Jul 09 18:05:11.827930 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.827909    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 18:05:11.828022 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.827953    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 18:05:11.832122 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.831377    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b56mt"]
Jul 09 18:05:11.832122 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.832030    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"
Jul 09 18:05:11.835907 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.835886    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 09 18:05:11.837929 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.836547    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:05:11.837929 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.836961    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gqm28\""
Jul 09 18:05:11.837929 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.837173    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 09 18:05:11.839512 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.839487    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"]
Jul 09 18:05:11.839648 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.839626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b56mt"
Jul 09 18:05:11.842730 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.842708    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-pzjcd"]
Jul 09 18:05:11.842961 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.842833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:05:11.843359 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.843341    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 18:05:11.843656 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.843569    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 18:05:11.843861 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.843843    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 18:05:11.844010 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.843992    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bf4m7\""
Jul 09 18:05:11.846029 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.846008    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pf6x5"]
Jul 09 18:05:11.846134 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.846045    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-ccd5k"]
Jul 09 18:05:11.846134 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.846071    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"]
Jul 09 18:05:11.847455 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.847436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzjcd"
Jul 09 18:05:11.847881 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.847862    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7hp5r\""
Jul 09 18:05:11.848514 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.848494    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 09 18:05:11.849027 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.848590    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 09 18:05:11.851955 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.851934    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tmgbd\""
Jul 09 18:05:11.852629 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.852147    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:05:11.852629 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.852620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"
Jul 09 18:05:11.853084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.851958    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:05:11.854062 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.854035    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"]
Jul 09 18:05:11.859060 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.859035    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b45b454c8-r66xm"]
Jul 09 18:05:11.860102 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.860075    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:11.860694 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.860680    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 18:05:11.861025 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.860706    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 18:05:11.862560 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.862539    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"]
Jul 09 18:05:11.862660 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.862587    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-pzjcd"]
Jul 09 18:05:11.862660 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.862612    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"]
Jul 09 18:05:11.862660 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.862639    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p7zkv"]
Jul 09 18:05:11.862856 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.862786    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b45b454c8-r66xm"
Jul 09 18:05:11.863009 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.862989    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"]
Jul 09 18:05:11.863323 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.863304    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 09 18:05:11.864639 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.864620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 18:05:11.867816 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.867771    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 18:05:11.867928 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.867859    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 18:05:11.868011 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.867984    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 18:05:11.868130 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.868113    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 18:05:11.868330 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.868311    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-s9d7f\""
Jul 09 18:05:11.868493 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.868474    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 09 18:05:11.882727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.882700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvwp8\" (UniqueName: \"kubernetes.io/projected/1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e-kube-api-access-lvwp8\") pod \"kube-storage-version-migrator-operator-799cb7f958-54gvz\" (UID: \"1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz"
Jul 09 18:05:11.882948 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.882923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-image-registry-private-configuration\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.883132 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppqmp\" (UniqueName: \"kubernetes.io/projected/7bb277a6-db3b-46bb-b1d1-ffca6119ec59-kube-api-access-ppqmp\") pod \"volume-data-source-validator-78495597f6-pf6x5\" (UID: \"7bb277a6-db3b-46bb-b1d1-ffca6119ec59\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pf6x5"
Jul 09 18:05:11.883290 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-bound-sa-token\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.883413 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/806524a3-c51c-49e3-822e-c04ed12d98cd-trusted-ca\") pod \"console-operator-7866497494-kln87\" (UID: \"806524a3-c51c-49e3-822e-c04ed12d98cd\") " pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:11.883636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-trusted-ca\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.883636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k56dj\" (UniqueName: \"kubernetes.io/projected/e815768d-07d5-486e-81c3-d29a321181a0-kube-api-access-k56dj\") pod \"service-ca-operator-7695fc6c8c-87nrw\" (UID: \"e815768d-07d5-486e-81c3-d29a321181a0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw"
Jul 09 18:05:11.883636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-default-certificate\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:11.883636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:11.883881 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-54gvz\" (UID: \"1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz"
Jul 09 18:05:11.883881 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1d25c758-d9bb-4113-a71d-de846c62e5ef-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:11.883881 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:11.883881 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/806524a3-c51c-49e3-822e-c04ed12d98cd-config\") pod \"console-operator-7866497494-kln87\" (UID: \"806524a3-c51c-49e3-822e-c04ed12d98cd\") " pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:11.883881 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e815768d-07d5-486e-81c3-d29a321181a0-config\") pod \"service-ca-operator-7695fc6c8c-87nrw\" (UID: \"e815768d-07d5-486e-81c3-d29a321181a0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw"
Jul 09 18:05:11.884112 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e815768d-07d5-486e-81c3-d29a321181a0-serving-cert\") pod \"service-ca-operator-7695fc6c8c-87nrw\" (UID: \"e815768d-07d5-486e-81c3-d29a321181a0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw"
Jul 09 18:05:11.884112 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.884112 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.883979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-459qd\" (UniqueName: \"kubernetes.io/projected/1d25c758-d9bb-4113-a71d-de846c62e5ef-kube-api-access-459qd\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:11.884112 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.884020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gkx7\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-kube-api-access-6gkx7\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.884112 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.884047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:11.884345 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.884112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-certificates\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.884345 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.884147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-installation-pull-secrets\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.884345 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.884186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-ca-trust-extracted\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.884345 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.884231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz7nf\" (UniqueName: \"kubernetes.io/projected/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-kube-api-access-sz7nf\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:11.884345 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.884260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc485\" (UniqueName: \"kubernetes.io/projected/806524a3-c51c-49e3-822e-c04ed12d98cd-kube-api-access-fc485\") pod \"console-operator-7866497494-kln87\" (UID: \"806524a3-c51c-49e3-822e-c04ed12d98cd\") " pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:11.884345 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.884309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-stats-auth\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:11.884706 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.884355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-54gvz\" (UID: \"1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz"
Jul 09 18:05:11.884706 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.884417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/806524a3-c51c-49e3-822e-c04ed12d98cd-serving-cert\") pod \"console-operator-7866497494-kln87\" (UID: \"806524a3-c51c-49e3-822e-c04ed12d98cd\") " pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:11.885327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.885310    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b45b454c8-r66xm"]
Jul 09 18:05:11.886451 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.886431    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"]
Jul 09 18:05:11.887352 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.887331    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b56mt"]
Jul 09 18:05:11.985308 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-stats-auth\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:11.985477 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9zf8g\" (UID: \"fccda6c1-c857-41be-a4fd-198cbd82a2c5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:05:11.985477 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shfw9\" (UniqueName: \"kubernetes.io/projected/2c211d21-6c7a-4c79-a433-751b85d263b4-kube-api-access-shfw9\") pod \"ingress-canary-b56mt\" (UID: \"2c211d21-6c7a-4c79-a433-751b85d263b4\") " pod="openshift-ingress-canary/ingress-canary-b56mt"
Jul 09 18:05:11.985477 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/12137a00-1ea6-420d-84be-97efa63bffe8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:11.985477 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-54gvz\" (UID: \"1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz"
Jul 09 18:05:11.985477 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/806524a3-c51c-49e3-822e-c04ed12d98cd-serving-cert\") pod \"console-operator-7866497494-kln87\" (UID: \"806524a3-c51c-49e3-822e-c04ed12d98cd\") " pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:11.985692 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/12137a00-1ea6-420d-84be-97efa63bffe8-ca\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:11.985692 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c211d21-6c7a-4c79-a433-751b85d263b4-cert\") pod \"ingress-canary-b56mt\" (UID: \"2c211d21-6c7a-4c79-a433-751b85d263b4\") " pod="openshift-ingress-canary/ingress-canary-b56mt"
Jul 09 18:05:11.985692 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-tmp\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:11.985692 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwpsc\" (UniqueName: \"kubernetes.io/projected/3b4a856f-2c4e-45b0-8d6f-25cce65d42a0-kube-api-access-fwpsc\") pod \"network-check-source-869dcfcdcd-pzjcd\" (UID: \"3b4a856f-2c4e-45b0-8d6f-25cce65d42a0\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzjcd"
Jul 09 18:05:11.985692 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6hfm\" (UniqueName: \"kubernetes.io/projected/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-kube-api-access-z6hfm\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:11.985692 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvwp8\" (UniqueName: \"kubernetes.io/projected/1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e-kube-api-access-lvwp8\") pod \"kube-storage-version-migrator-operator-799cb7f958-54gvz\" (UID: \"1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz"
Jul 09 18:05:11.985692 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/603013ff-71ad-4016-b7cb-8bdf4f65448d-tmp\") pod \"klusterlet-addon-workmgr-77c5d99496-vl2z9\" (UID: \"603013ff-71ad-4016-b7cb-8bdf4f65448d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"
Jul 09 18:05:11.986038 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-image-registry-private-configuration\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.986038 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppqmp\" (UniqueName: \"kubernetes.io/projected/7bb277a6-db3b-46bb-b1d1-ffca6119ec59-kube-api-access-ppqmp\") pod \"volume-data-source-validator-78495597f6-pf6x5\" (UID: \"7bb277a6-db3b-46bb-b1d1-ffca6119ec59\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pf6x5"
Jul 09 18:05:11.986038 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-bound-sa-token\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.986038 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/12137a00-1ea6-420d-84be-97efa63bffe8-hub\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:11.986038 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/12137a00-1ea6-420d-84be-97efa63bffe8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:11.986038 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2wnng\" (UID: \"4b9b442e-e7e5-4b54-bd16-5141660c77ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"
Jul 09 18:05:11.986038 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc66w\" (UniqueName: \"kubernetes.io/projected/4b9b442e-e7e5-4b54-bd16-5141660c77ce-kube-api-access-dc66w\") pod \"cluster-samples-operator-65db858894-2wnng\" (UID: \"4b9b442e-e7e5-4b54-bd16-5141660c77ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"
Jul 09 18:05:11.986038 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.985963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:11.986038 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/806524a3-c51c-49e3-822e-c04ed12d98cd-trusted-ca\") pod \"console-operator-7866497494-kln87\" (UID: \"806524a3-c51c-49e3-822e-c04ed12d98cd\") " pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:11.986038 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/603013ff-71ad-4016-b7cb-8bdf4f65448d-klusterlet-config\") pod \"klusterlet-addon-workmgr-77c5d99496-vl2z9\" (UID: \"603013ff-71ad-4016-b7cb-8bdf4f65448d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"
Jul 09 18:05:11.986485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:11.986485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-trusted-ca\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.986485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k56dj\" (UniqueName: \"kubernetes.io/projected/e815768d-07d5-486e-81c3-d29a321181a0-kube-api-access-k56dj\") pod \"service-ca-operator-7695fc6c8c-87nrw\" (UID: \"e815768d-07d5-486e-81c3-d29a321181a0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw"
Jul 09 18:05:11.986485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fccda6c1-c857-41be-a4fd-198cbd82a2c5-nginx-conf\") pod \"networking-console-plugin-6cf856758c-9zf8g\" (UID: \"fccda6c1-c857-41be-a4fd-198cbd82a2c5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:05:11.986485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-default-certificate\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:11.986485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:11.986485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2psl4\" (UniqueName: \"kubernetes.io/projected/603013ff-71ad-4016-b7cb-8bdf4f65448d-kube-api-access-2psl4\") pod \"klusterlet-addon-workmgr-77c5d99496-vl2z9\" (UID: \"603013ff-71ad-4016-b7cb-8bdf4f65448d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"
Jul 09 18:05:11.986485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/122683df-b7fe-4c5a-9974-987713ba1ee2-tmp-dir\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:11.986485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12137a00-1ea6-420d-84be-97efa63bffe8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:11.986929 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.986521    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle podName:c5c938ef-4491-4fc7-aacb-02fa8e3ce61b nodeName:}" failed. No retries permitted until 2026-07-09 18:05:12.48650151 +0000 UTC m=+35.673479258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle") pod "router-default-675d564c74-s5fsx" (UID: "c5c938ef-4491-4fc7-aacb-02fa8e3ce61b") : configmap references non-existent config key: service-ca.crt
Jul 09 18:05:11.986929 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-54gvz\" (UID: \"1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz"
Jul 09 18:05:11.986929 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:11.986929 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1d25c758-d9bb-4113-a71d-de846c62e5ef-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:11.986929 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:11.986929 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/924ac8cc-82c8-40e7-995d-e10eb77aa558-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b45b454c8-r66xm\" (UID: \"924ac8cc-82c8-40e7-995d-e10eb77aa558\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b45b454c8-r66xm"
Jul 09 18:05:11.986929 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.986837    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:05:11.986929 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.986885    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls podName:1d25c758-d9bb-4113-a71d-de846c62e5ef nodeName:}" failed. No retries permitted until 2026-07-09 18:05:12.486870383 +0000 UTC m=+35.673848161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-cjt5m" (UID: "1d25c758-d9bb-4113-a71d-de846c62e5ef") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:05:11.986929 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/806524a3-c51c-49e3-822e-c04ed12d98cd-config\") pod \"console-operator-7866497494-kln87\" (UID: \"806524a3-c51c-49e3-822e-c04ed12d98cd\") " pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:11.987348 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e815768d-07d5-486e-81c3-d29a321181a0-config\") pod \"service-ca-operator-7695fc6c8c-87nrw\" (UID: \"e815768d-07d5-486e-81c3-d29a321181a0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw"
Jul 09 18:05:11.987348 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktm82\" (UniqueName: \"kubernetes.io/projected/12137a00-1ea6-420d-84be-97efa63bffe8-kube-api-access-ktm82\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:11.987348 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.986990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4bnp\" (UniqueName: \"kubernetes.io/projected/924ac8cc-82c8-40e7-995d-e10eb77aa558-kube-api-access-h4bnp\") pod \"managed-serviceaccount-addon-agent-6b45b454c8-r66xm\" (UID: \"924ac8cc-82c8-40e7-995d-e10eb77aa558\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b45b454c8-r66xm"
Jul 09 18:05:11.987348 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.987012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/122683df-b7fe-4c5a-9974-987713ba1ee2-config-volume\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:11.987348 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.987034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zwxd\" (UniqueName: \"kubernetes.io/projected/122683df-b7fe-4c5a-9974-987713ba1ee2-kube-api-access-7zwxd\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:11.987348 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.987059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-serving-cert\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:11.987348 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.987120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e815768d-07d5-486e-81c3-d29a321181a0-serving-cert\") pod \"service-ca-operator-7695fc6c8c-87nrw\" (UID: \"e815768d-07d5-486e-81c3-d29a321181a0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw"
Jul 09 18:05:11.987348 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.987146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.987348 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.987176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-459qd\" (UniqueName: \"kubernetes.io/projected/1d25c758-d9bb-4113-a71d-de846c62e5ef-kube-api-access-459qd\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:11.987348 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.987204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gkx7\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-kube-api-access-6gkx7\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.987348 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.987227    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:11.987348 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.987251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-snapshots\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:11.987917 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.987472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-trusted-ca\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.987917 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.987478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1d25c758-d9bb-4113-a71d-de846c62e5ef-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:11.987917 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.987763    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-54gvz\" (UID: \"1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz"
Jul 09 18:05:11.988304 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.988192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/806524a3-c51c-49e3-822e-c04ed12d98cd-config\") pod \"console-operator-7866497494-kln87\" (UID: \"806524a3-c51c-49e3-822e-c04ed12d98cd\") " pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:11.989470 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.988557    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:05:11.989470 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.988577    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6846c96d87-7mhdw: secret "image-registry-tls" not found
Jul 09 18:05:11.989470 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.988666    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls podName:f6363dd9-59ec-4cbd-ae82-a9e896a0814c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:12.488647865 +0000 UTC m=+35.675625614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls") pod "image-registry-6846c96d87-7mhdw" (UID: "f6363dd9-59ec-4cbd-ae82-a9e896a0814c") : secret "image-registry-tls" not found
Jul 09 18:05:11.989470 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.988706    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:05:11.989470 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.988716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-certificates\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.989470 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.988748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-installation-pull-secrets\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.989470 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:11.988766    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs podName:c5c938ef-4491-4fc7-aacb-02fa8e3ce61b nodeName:}" failed. No retries permitted until 2026-07-09 18:05:12.488749971 +0000 UTC m=+35.675727721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs") pod "router-default-675d564c74-s5fsx" (UID: "c5c938ef-4491-4fc7-aacb-02fa8e3ce61b") : secret "router-metrics-certs-default" not found
Jul 09 18:05:11.989470 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.988772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e815768d-07d5-486e-81c3-d29a321181a0-config\") pod \"service-ca-operator-7695fc6c8c-87nrw\" (UID: \"e815768d-07d5-486e-81c3-d29a321181a0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw"
Jul 09 18:05:11.989470 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.988844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-ca-trust-extracted\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.989470 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.988933    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/806524a3-c51c-49e3-822e-c04ed12d98cd-trusted-ca\") pod \"console-operator-7866497494-kln87\" (UID: \"806524a3-c51c-49e3-822e-c04ed12d98cd\") " pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:11.989470 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.989006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz7nf\" (UniqueName: \"kubernetes.io/projected/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-kube-api-access-sz7nf\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:11.989470 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.989060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc485\" (UniqueName: \"kubernetes.io/projected/806524a3-c51c-49e3-822e-c04ed12d98cd-kube-api-access-fc485\") pod \"console-operator-7866497494-kln87\" (UID: \"806524a3-c51c-49e3-822e-c04ed12d98cd\") " pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:11.989470 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.989105    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-ca-trust-extracted\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.989470 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.989246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-certificates\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.991146 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.991117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-image-registry-private-configuration\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.991377 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.991322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e815768d-07d5-486e-81c3-d29a321181a0-serving-cert\") pod \"service-ca-operator-7695fc6c8c-87nrw\" (UID: \"e815768d-07d5-486e-81c3-d29a321181a0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw"
Jul 09 18:05:11.991750 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.991724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/806524a3-c51c-49e3-822e-c04ed12d98cd-serving-cert\") pod \"console-operator-7866497494-kln87\" (UID: \"806524a3-c51c-49e3-822e-c04ed12d98cd\") " pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:11.991889 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.991868    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-installation-pull-secrets\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:11.992292 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.992251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-default-certificate\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:11.992382 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.992295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-54gvz\" (UID: \"1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz"
Jul 09 18:05:11.992603 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:11.992583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-stats-auth\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:12.001853 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.001757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k56dj\" (UniqueName: \"kubernetes.io/projected/e815768d-07d5-486e-81c3-d29a321181a0-kube-api-access-k56dj\") pod \"service-ca-operator-7695fc6c8c-87nrw\" (UID: \"e815768d-07d5-486e-81c3-d29a321181a0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw"
Jul 09 18:05:12.002397 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.002365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppqmp\" (UniqueName: \"kubernetes.io/projected/7bb277a6-db3b-46bb-b1d1-ffca6119ec59-kube-api-access-ppqmp\") pod \"volume-data-source-validator-78495597f6-pf6x5\" (UID: \"7bb277a6-db3b-46bb-b1d1-ffca6119ec59\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pf6x5"
Jul 09 18:05:12.003645 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.003617    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvwp8\" (UniqueName: \"kubernetes.io/projected/1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e-kube-api-access-lvwp8\") pod \"kube-storage-version-migrator-operator-799cb7f958-54gvz\" (UID: \"1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz"
Jul 09 18:05:12.003645 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.003633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-bound-sa-token\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:12.014307 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.014272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz7nf\" (UniqueName: \"kubernetes.io/projected/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-kube-api-access-sz7nf\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:12.014952 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.014906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gkx7\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-kube-api-access-6gkx7\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:12.015498 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.015476    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc485\" (UniqueName: \"kubernetes.io/projected/806524a3-c51c-49e3-822e-c04ed12d98cd-kube-api-access-fc485\") pod \"console-operator-7866497494-kln87\" (UID: \"806524a3-c51c-49e3-822e-c04ed12d98cd\") " pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:12.016058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.016032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-459qd\" (UniqueName: \"kubernetes.io/projected/1d25c758-d9bb-4113-a71d-de846c62e5ef-kube-api-access-459qd\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:12.057962 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.057920    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz"
Jul 09 18:05:12.089899 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.089860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/12137a00-1ea6-420d-84be-97efa63bffe8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:12.089899 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.089907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/12137a00-1ea6-420d-84be-97efa63bffe8-ca\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:12.090152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.089933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c211d21-6c7a-4c79-a433-751b85d263b4-cert\") pod \"ingress-canary-b56mt\" (UID: \"2c211d21-6c7a-4c79-a433-751b85d263b4\") " pod="openshift-ingress-canary/ingress-canary-b56mt"
Jul 09 18:05:12.090152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.089958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-tmp\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:12.090152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.089984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpsc\" (UniqueName: \"kubernetes.io/projected/3b4a856f-2c4e-45b0-8d6f-25cce65d42a0-kube-api-access-fwpsc\") pod \"network-check-source-869dcfcdcd-pzjcd\" (UID: \"3b4a856f-2c4e-45b0-8d6f-25cce65d42a0\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzjcd"
Jul 09 18:05:12.090152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6hfm\" (UniqueName: \"kubernetes.io/projected/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-kube-api-access-z6hfm\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:12.090152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/603013ff-71ad-4016-b7cb-8bdf4f65448d-tmp\") pod \"klusterlet-addon-workmgr-77c5d99496-vl2z9\" (UID: \"603013ff-71ad-4016-b7cb-8bdf4f65448d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"
Jul 09 18:05:12.090152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/12137a00-1ea6-420d-84be-97efa63bffe8-hub\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:12.090152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/12137a00-1ea6-420d-84be-97efa63bffe8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:12.090152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2wnng\" (UID: \"4b9b442e-e7e5-4b54-bd16-5141660c77ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"
Jul 09 18:05:12.090152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc66w\" (UniqueName: \"kubernetes.io/projected/4b9b442e-e7e5-4b54-bd16-5141660c77ce-kube-api-access-dc66w\") pod \"cluster-samples-operator-65db858894-2wnng\" (UID: \"4b9b442e-e7e5-4b54-bd16-5141660c77ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"
Jul 09 18:05:12.090562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:12.090562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/603013ff-71ad-4016-b7cb-8bdf4f65448d-klusterlet-config\") pod \"klusterlet-addon-workmgr-77c5d99496-vl2z9\" (UID: \"603013ff-71ad-4016-b7cb-8bdf4f65448d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"
Jul 09 18:05:12.090562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:12.090562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fccda6c1-c857-41be-a4fd-198cbd82a2c5-nginx-conf\") pod \"networking-console-plugin-6cf856758c-9zf8g\" (UID: \"fccda6c1-c857-41be-a4fd-198cbd82a2c5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:05:12.090562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2psl4\" (UniqueName: \"kubernetes.io/projected/603013ff-71ad-4016-b7cb-8bdf4f65448d-kube-api-access-2psl4\") pod \"klusterlet-addon-workmgr-77c5d99496-vl2z9\" (UID: \"603013ff-71ad-4016-b7cb-8bdf4f65448d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"
Jul 09 18:05:12.090562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/122683df-b7fe-4c5a-9974-987713ba1ee2-tmp-dir\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:12.090562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12137a00-1ea6-420d-84be-97efa63bffe8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:12.090562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090390    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-tmp\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:12.090562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:12.090562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/924ac8cc-82c8-40e7-995d-e10eb77aa558-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b45b454c8-r66xm\" (UID: \"924ac8cc-82c8-40e7-995d-e10eb77aa558\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b45b454c8-r66xm"
Jul 09 18:05:12.090562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktm82\" (UniqueName: \"kubernetes.io/projected/12137a00-1ea6-420d-84be-97efa63bffe8-kube-api-access-ktm82\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:12.090562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4bnp\" (UniqueName: \"kubernetes.io/projected/924ac8cc-82c8-40e7-995d-e10eb77aa558-kube-api-access-h4bnp\") pod \"managed-serviceaccount-addon-agent-6b45b454c8-r66xm\" (UID: \"924ac8cc-82c8-40e7-995d-e10eb77aa558\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b45b454c8-r66xm"
Jul 09 18:05:12.090562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/122683df-b7fe-4c5a-9974-987713ba1ee2-config-volume\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:12.091184 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zwxd\" (UniqueName: \"kubernetes.io/projected/122683df-b7fe-4c5a-9974-987713ba1ee2-kube-api-access-7zwxd\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:12.091184 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-serving-cert\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:12.091184 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/12137a00-1ea6-420d-84be-97efa63bffe8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:12.091184 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-snapshots\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:12.091184 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9zf8g\" (UID: \"fccda6c1-c857-41be-a4fd-198cbd82a2c5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:05:12.091184 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.090858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shfw9\" (UniqueName: \"kubernetes.io/projected/2c211d21-6c7a-4c79-a433-751b85d263b4-kube-api-access-shfw9\") pod \"ingress-canary-b56mt\" (UID: \"2c211d21-6c7a-4c79-a433-751b85d263b4\") " pod="openshift-ingress-canary/ingress-canary-b56mt"
Jul 09 18:05:12.091481 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.091249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:12.091481 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.091383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/603013ff-71ad-4016-b7cb-8bdf4f65448d-tmp\") pod \"klusterlet-addon-workmgr-77c5d99496-vl2z9\" (UID: \"603013ff-71ad-4016-b7cb-8bdf4f65448d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"
Jul 09 18:05:12.091579 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.091562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fccda6c1-c857-41be-a4fd-198cbd82a2c5-nginx-conf\") pod \"networking-console-plugin-6cf856758c-9zf8g\" (UID: \"fccda6c1-c857-41be-a4fd-198cbd82a2c5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.091668    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.091709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-snapshots\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.091736    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls podName:122683df-b7fe-4c5a-9974-987713ba1ee2 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:12.591716048 +0000 UTC m=+35.778693806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls") pod "dns-default-p7zkv" (UID: "122683df-b7fe-4c5a-9974-987713ba1ee2") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.091678    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/122683df-b7fe-4c5a-9974-987713ba1ee2-tmp-dir\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.091779    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.091841    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.091856    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert podName:fccda6c1-c857-41be-a4fd-198cbd82a2c5 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:12.591839949 +0000 UTC m=+35.778817697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-9zf8g" (UID: "fccda6c1-c857-41be-a4fd-198cbd82a2c5") : secret "networking-console-plugin-cert" not found
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.091897    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls podName:4b9b442e-e7e5-4b54-bd16-5141660c77ce nodeName:}" failed. No retries permitted until 2026-07-09 18:05:12.591882245 +0000 UTC m=+35.778860014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls") pod "cluster-samples-operator-65db858894-2wnng" (UID: "4b9b442e-e7e5-4b54-bd16-5141660c77ce") : secret "samples-operator-tls" not found
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.093192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/122683df-b7fe-4c5a-9974-987713ba1ee2-config-volume\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.094021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/12137a00-1ea6-420d-84be-97efa63bffe8-ca\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.094236    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.094412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-serving-cert\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.094760    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12137a00-1ea6-420d-84be-97efa63bffe8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.094977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/12137a00-1ea6-420d-84be-97efa63bffe8-hub\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:12.095369 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.095075    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:12.096427 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.095477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/12137a00-1ea6-420d-84be-97efa63bffe8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:12.096427 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.096232    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c211d21-6c7a-4c79-a433-751b85d263b4-cert\") pod \"ingress-canary-b56mt\" (UID: \"2c211d21-6c7a-4c79-a433-751b85d263b4\") " pod="openshift-ingress-canary/ingress-canary-b56mt"
Jul 09 18:05:12.097095 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.097074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/924ac8cc-82c8-40e7-995d-e10eb77aa558-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b45b454c8-r66xm\" (UID: \"924ac8cc-82c8-40e7-995d-e10eb77aa558\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b45b454c8-r66xm"
Jul 09 18:05:12.097576 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.097552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/603013ff-71ad-4016-b7cb-8bdf4f65448d-klusterlet-config\") pod \"klusterlet-addon-workmgr-77c5d99496-vl2z9\" (UID: \"603013ff-71ad-4016-b7cb-8bdf4f65448d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"
Jul 09 18:05:12.102787 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.102759    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw"
Jul 09 18:05:12.105906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.105843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc66w\" (UniqueName: \"kubernetes.io/projected/4b9b442e-e7e5-4b54-bd16-5141660c77ce-kube-api-access-dc66w\") pod \"cluster-samples-operator-65db858894-2wnng\" (UID: \"4b9b442e-e7e5-4b54-bd16-5141660c77ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"
Jul 09 18:05:12.106009 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.105936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6hfm\" (UniqueName: \"kubernetes.io/projected/5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b-kube-api-access-z6hfm\") pod \"insights-operator-68d9dd94cf-ccd5k\" (UID: \"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b\") " pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:12.106789 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.106769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zwxd\" (UniqueName: \"kubernetes.io/projected/122683df-b7fe-4c5a-9974-987713ba1ee2-kube-api-access-7zwxd\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:12.107036 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.107012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktm82\" (UniqueName: \"kubernetes.io/projected/12137a00-1ea6-420d-84be-97efa63bffe8-kube-api-access-ktm82\") pod \"cluster-proxy-proxy-agent-74f75ddc54-cncf6\" (UID: \"12137a00-1ea6-420d-84be-97efa63bffe8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:12.108015 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.107970    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4bnp\" (UniqueName: \"kubernetes.io/projected/924ac8cc-82c8-40e7-995d-e10eb77aa558-kube-api-access-h4bnp\") pod \"managed-serviceaccount-addon-agent-6b45b454c8-r66xm\" (UID: \"924ac8cc-82c8-40e7-995d-e10eb77aa558\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b45b454c8-r66xm"
Jul 09 18:05:12.108855 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.108819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shfw9\" (UniqueName: \"kubernetes.io/projected/2c211d21-6c7a-4c79-a433-751b85d263b4-kube-api-access-shfw9\") pod \"ingress-canary-b56mt\" (UID: \"2c211d21-6c7a-4c79-a433-751b85d263b4\") " pod="openshift-ingress-canary/ingress-canary-b56mt"
Jul 09 18:05:12.109088 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.109064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2psl4\" (UniqueName: \"kubernetes.io/projected/603013ff-71ad-4016-b7cb-8bdf4f65448d-kube-api-access-2psl4\") pod \"klusterlet-addon-workmgr-77c5d99496-vl2z9\" (UID: \"603013ff-71ad-4016-b7cb-8bdf4f65448d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"
Jul 09 18:05:12.109441 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.109410    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pf6x5"
Jul 09 18:05:12.109896 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.109874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwpsc\" (UniqueName: \"kubernetes.io/projected/3b4a856f-2c4e-45b0-8d6f-25cce65d42a0-kube-api-access-fwpsc\") pod \"network-check-source-869dcfcdcd-pzjcd\" (UID: \"3b4a856f-2c4e-45b0-8d6f-25cce65d42a0\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzjcd"
Jul 09 18:05:12.116568 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.116543    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k"
Jul 09 18:05:12.162636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.162591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b56mt"
Jul 09 18:05:12.180211 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.180177    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzjcd"
Jul 09 18:05:12.191442 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.191411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"
Jul 09 18:05:12.213634 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.213598    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:05:12.220669 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.220642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b45b454c8-r66xm"
Jul 09 18:05:12.397680 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.397602    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:05:12.397862 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.397602    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:05:12.397862 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.397602    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:05:12.401766 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.401745    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhwtc\""
Jul 09 18:05:12.402038 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.401963    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:12.402747 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.402070    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:12.402747 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.402485    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vcghf\""
Jul 09 18:05:12.495091 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.495050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:12.495261 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.495109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:12.495261 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.495144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:12.495261 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.495169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:12.495441 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.495275    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle podName:c5c938ef-4491-4fc7-aacb-02fa8e3ce61b nodeName:}" failed. No retries permitted until 2026-07-09 18:05:13.495248404 +0000 UTC m=+36.682226156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle") pod "router-default-675d564c74-s5fsx" (UID: "c5c938ef-4491-4fc7-aacb-02fa8e3ce61b") : configmap references non-existent config key: service-ca.crt
Jul 09 18:05:12.495441 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.495286    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:05:12.495441 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.495296    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:05:12.495441 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.495313    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:05:12.495441 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.495331    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6846c96d87-7mhdw: secret "image-registry-tls" not found
Jul 09 18:05:12.495441 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.495346    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs podName:c5c938ef-4491-4fc7-aacb-02fa8e3ce61b nodeName:}" failed. No retries permitted until 2026-07-09 18:05:13.495329862 +0000 UTC m=+36.682307630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs") pod "router-default-675d564c74-s5fsx" (UID: "c5c938ef-4491-4fc7-aacb-02fa8e3ce61b") : secret "router-metrics-certs-default" not found
Jul 09 18:05:12.495441 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.495392    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls podName:f6363dd9-59ec-4cbd-ae82-a9e896a0814c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:13.495376292 +0000 UTC m=+36.682354042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls") pod "image-registry-6846c96d87-7mhdw" (UID: "f6363dd9-59ec-4cbd-ae82-a9e896a0814c") : secret "image-registry-tls" not found
Jul 09 18:05:12.495441 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.495413    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls podName:1d25c758-d9bb-4113-a71d-de846c62e5ef nodeName:}" failed. No retries permitted until 2026-07-09 18:05:13.495402324 +0000 UTC m=+36.682380071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-cjt5m" (UID: "1d25c758-d9bb-4113-a71d-de846c62e5ef") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:05:12.596545 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.596496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9zf8g\" (UID: \"fccda6c1-c857-41be-a4fd-198cbd82a2c5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:05:12.596760 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.596604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2wnng\" (UID: \"4b9b442e-e7e5-4b54-bd16-5141660c77ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"
Jul 09 18:05:12.596760 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:12.596633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:12.596760 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.596680    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:05:12.596760 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.596759    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert podName:fccda6c1-c857-41be-a4fd-198cbd82a2c5 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:13.596736504 +0000 UTC m=+36.783714254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-9zf8g" (UID: "fccda6c1-c857-41be-a4fd-198cbd82a2c5") : secret "networking-console-plugin-cert" not found
Jul 09 18:05:12.597013 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.596772    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:05:12.597013 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.596781    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:12.597013 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.596850    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls podName:4b9b442e-e7e5-4b54-bd16-5141660c77ce nodeName:}" failed. No retries permitted until 2026-07-09 18:05:13.596834332 +0000 UTC m=+36.783812084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls") pod "cluster-samples-operator-65db858894-2wnng" (UID: "4b9b442e-e7e5-4b54-bd16-5141660c77ce") : secret "samples-operator-tls" not found
Jul 09 18:05:12.597013 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:12.596880    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls podName:122683df-b7fe-4c5a-9974-987713ba1ee2 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:13.596861855 +0000 UTC m=+36.783839604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls") pod "dns-default-p7zkv" (UID: "122683df-b7fe-4c5a-9974-987713ba1ee2") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:13.530201 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.522314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:13.530201 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.522620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:13.530201 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.522667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:13.530201 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.522715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:13.530201 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:13.522963    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:05:13.530201 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:13.523035    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs podName:c5c938ef-4491-4fc7-aacb-02fa8e3ce61b nodeName:}" failed. No retries permitted until 2026-07-09 18:05:15.523013896 +0000 UTC m=+38.709991664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs") pod "router-default-675d564c74-s5fsx" (UID: "c5c938ef-4491-4fc7-aacb-02fa8e3ce61b") : secret "router-metrics-certs-default" not found
Jul 09 18:05:13.530201 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:13.523735    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle podName:c5c938ef-4491-4fc7-aacb-02fa8e3ce61b nodeName:}" failed. No retries permitted until 2026-07-09 18:05:15.523713545 +0000 UTC m=+38.710691298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle") pod "router-default-675d564c74-s5fsx" (UID: "c5c938ef-4491-4fc7-aacb-02fa8e3ce61b") : configmap references non-existent config key: service-ca.crt
Jul 09 18:05:13.530201 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:13.523841    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:05:13.530201 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:13.523882    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls podName:1d25c758-d9bb-4113-a71d-de846c62e5ef nodeName:}" failed. No retries permitted until 2026-07-09 18:05:15.523869103 +0000 UTC m=+38.710846855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-cjt5m" (UID: "1d25c758-d9bb-4113-a71d-de846c62e5ef") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:05:13.530201 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:13.523947    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:05:13.530201 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:13.523958    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6846c96d87-7mhdw: secret "image-registry-tls" not found
Jul 09 18:05:13.530201 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:13.523990    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls podName:f6363dd9-59ec-4cbd-ae82-a9e896a0814c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:15.52398028 +0000 UTC m=+38.710958040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls") pod "image-registry-6846c96d87-7mhdw" (UID: "f6363dd9-59ec-4cbd-ae82-a9e896a0814c") : secret "image-registry-tls" not found
Jul 09 18:05:13.586757 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.586498    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b56mt"]
Jul 09 18:05:13.591071 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.589884    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"]
Jul 09 18:05:13.591071 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.589944    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz"]
Jul 09 18:05:13.593255 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:05:13.592588    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6be2b5_a1e6_407a_bbf7_3aecabb2b99e.slice/crio-5cd80a79c2b3fbaa9468a04d8242f2931bfc47e4894f03b3374bfb1ca2b5d58d WatchSource:0}: Error finding container 5cd80a79c2b3fbaa9468a04d8242f2931bfc47e4894f03b3374bfb1ca2b5d58d: Status 404 returned error can't find the container with id 5cd80a79c2b3fbaa9468a04d8242f2931bfc47e4894f03b3374bfb1ca2b5d58d
Jul 09 18:05:13.635595 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.629627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9zf8g\" (UID: \"fccda6c1-c857-41be-a4fd-198cbd82a2c5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:05:13.635595 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.629735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2wnng\" (UID: \"4b9b442e-e7e5-4b54-bd16-5141660c77ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"
Jul 09 18:05:13.635595 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.629765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:13.635595 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:13.629993    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:13.635595 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:13.630070    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls podName:122683df-b7fe-4c5a-9974-987713ba1ee2 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:15.630050723 +0000 UTC m=+38.817028473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls") pod "dns-default-p7zkv" (UID: "122683df-b7fe-4c5a-9974-987713ba1ee2") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:13.635595 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:13.630522    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:05:13.635595 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:13.630575    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert podName:fccda6c1-c857-41be-a4fd-198cbd82a2c5 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:15.630558653 +0000 UTC m=+38.817536414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-9zf8g" (UID: "fccda6c1-c857-41be-a4fd-198cbd82a2c5") : secret "networking-console-plugin-cert" not found
Jul 09 18:05:13.635595 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:13.630645    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:05:13.635595 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:13.630680    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls podName:4b9b442e-e7e5-4b54-bd16-5141660c77ce nodeName:}" failed. No retries permitted until 2026-07-09 18:05:15.630668961 +0000 UTC m=+38.817646726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls") pod "cluster-samples-operator-65db858894-2wnng" (UID: "4b9b442e-e7e5-4b54-bd16-5141660c77ce") : secret "samples-operator-tls" not found
Jul 09 18:05:13.678480 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.678457    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-kln87"]
Jul 09 18:05:13.681404 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.681380    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b45b454c8-r66xm"]
Jul 09 18:05:13.693055 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.693030    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"]
Jul 09 18:05:13.703385 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.703316    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pf6x5"]
Jul 09 18:05:13.704318 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.704290    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw"]
Jul 09 18:05:13.710673 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.710653    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-ccd5k"]
Jul 09 18:05:13.711882 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:13.711602    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-pzjcd"]
Jul 09 18:05:13.714749 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:05:13.714718    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806524a3_c51c_49e3_822e_c04ed12d98cd.slice/crio-1c0d96b9ee63404fb1e2b4308c14eba285cbc5c48dcf2a8c5329281bb6a64c18 WatchSource:0}: Error finding container 1c0d96b9ee63404fb1e2b4308c14eba285cbc5c48dcf2a8c5329281bb6a64c18: Status 404 returned error can't find the container with id 1c0d96b9ee63404fb1e2b4308c14eba285cbc5c48dcf2a8c5329281bb6a64c18
Jul 09 18:05:13.715126 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:05:13.715101    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9404a1_b4e7_43e5_b1c5_6eec92b2df4b.slice/crio-034a211f2a6d34a9d60199e979145521d2a152beddffabc75dde1e7bf2281031 WatchSource:0}: Error finding container 034a211f2a6d34a9d60199e979145521d2a152beddffabc75dde1e7bf2281031: Status 404 returned error can't find the container with id 034a211f2a6d34a9d60199e979145521d2a152beddffabc75dde1e7bf2281031
Jul 09 18:05:13.716199 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:05:13.716172    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode815768d_07d5_486e_81c3_d29a321181a0.slice/crio-50892b8c11bcdb2797e5aef4ccffaa1b38b0039ff1e18183ea9ecc327c47f8fd WatchSource:0}: Error finding container 50892b8c11bcdb2797e5aef4ccffaa1b38b0039ff1e18183ea9ecc327c47f8fd: Status 404 returned error can't find the container with id 50892b8c11bcdb2797e5aef4ccffaa1b38b0039ff1e18183ea9ecc327c47f8fd
Jul 09 18:05:13.718239 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:05:13.718081    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb277a6_db3b_46bb_b1d1_ffca6119ec59.slice/crio-90f390199c573d4093ed9e4b3e9b6665d54c4927777d180ad1ead5dec764eea0 WatchSource:0}: Error finding container 90f390199c573d4093ed9e4b3e9b6665d54c4927777d180ad1ead5dec764eea0: Status 404 returned error can't find the container with id 90f390199c573d4093ed9e4b3e9b6665d54c4927777d180ad1ead5dec764eea0
Jul 09 18:05:13.721617 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:05:13.721152    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod924ac8cc_82c8_40e7_995d_e10eb77aa558.slice/crio-24bf7ff79477f79695be8b63d72da5a170b90449dd4df1af90c0cd6d82e786a3 WatchSource:0}: Error finding container 24bf7ff79477f79695be8b63d72da5a170b90449dd4df1af90c0cd6d82e786a3: Status 404 returned error can't find the container with id 24bf7ff79477f79695be8b63d72da5a170b90449dd4df1af90c0cd6d82e786a3
Jul 09 18:05:13.721617 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:05:13.721496    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4a856f_2c4e_45b0_8d6f_25cce65d42a0.slice/crio-27b7d955fc86f40d005b467ce142a36ba13a4b8caeb0719b824ce99496c81039 WatchSource:0}: Error finding container 27b7d955fc86f40d005b467ce142a36ba13a4b8caeb0719b824ce99496c81039: Status 404 returned error can't find the container with id 27b7d955fc86f40d005b467ce142a36ba13a4b8caeb0719b824ce99496c81039
Jul 09 18:05:13.722475 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:05:13.722443    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12137a00_1ea6_420d_84be_97efa63bffe8.slice/crio-0bf5c4b240342848467f5c74b4b452619da8ac0232a885546c037e1ea43703e3 WatchSource:0}: Error finding container 0bf5c4b240342848467f5c74b4b452619da8ac0232a885546c037e1ea43703e3: Status 404 returned error can't find the container with id 0bf5c4b240342848467f5c74b4b452619da8ac0232a885546c037e1ea43703e3
Jul 09 18:05:14.576598 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:14.576464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k" event={"ID":"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b","Type":"ContainerStarted","Data":"034a211f2a6d34a9d60199e979145521d2a152beddffabc75dde1e7bf2281031"}
Jul 09 18:05:14.579818 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:14.579743    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b45b454c8-r66xm" event={"ID":"924ac8cc-82c8-40e7-995d-e10eb77aa558","Type":"ContainerStarted","Data":"24bf7ff79477f79695be8b63d72da5a170b90449dd4df1af90c0cd6d82e786a3"}
Jul 09 18:05:14.591301 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:14.591161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw" event={"ID":"e815768d-07d5-486e-81c3-d29a321181a0","Type":"ContainerStarted","Data":"50892b8c11bcdb2797e5aef4ccffaa1b38b0039ff1e18183ea9ecc327c47f8fd"}
Jul 09 18:05:14.611100 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:14.611050    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-kln87" event={"ID":"806524a3-c51c-49e3-822e-c04ed12d98cd","Type":"ContainerStarted","Data":"1c0d96b9ee63404fb1e2b4308c14eba285cbc5c48dcf2a8c5329281bb6a64c18"}
Jul 09 18:05:14.619135 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:14.619079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pf6x5" event={"ID":"7bb277a6-db3b-46bb-b1d1-ffca6119ec59","Type":"ContainerStarted","Data":"90f390199c573d4093ed9e4b3e9b6665d54c4927777d180ad1ead5dec764eea0"}
Jul 09 18:05:14.629303 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:14.629262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6" event={"ID":"12137a00-1ea6-420d-84be-97efa63bffe8","Type":"ContainerStarted","Data":"0bf5c4b240342848467f5c74b4b452619da8ac0232a885546c037e1ea43703e3"}
Jul 09 18:05:14.634238 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:14.634197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b56mt" event={"ID":"2c211d21-6c7a-4c79-a433-751b85d263b4","Type":"ContainerStarted","Data":"4c43b6a8a02ae3c10f9a3e2460dbb7c7978ebef02c78cd5cf10cd5f7f2695dc2"}
Jul 09 18:05:14.644492 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:14.644445    2578 generic.go:356] "Generic (PLEG): container finished" podID="d9f223b1-0765-46bb-b8af-136743836d8a" containerID="04a0677144f7b49e5a31fecb14ddab50109afd00675cf1be96f7f9dbf8bb7c1a" exitCode=0
Jul 09 18:05:14.644652 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:14.644551    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jmw8" event={"ID":"d9f223b1-0765-46bb-b8af-136743836d8a","Type":"ContainerDied","Data":"04a0677144f7b49e5a31fecb14ddab50109afd00675cf1be96f7f9dbf8bb7c1a"}
Jul 09 18:05:14.658824 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:14.658767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzjcd" event={"ID":"3b4a856f-2c4e-45b0-8d6f-25cce65d42a0","Type":"ContainerStarted","Data":"27b7d955fc86f40d005b467ce142a36ba13a4b8caeb0719b824ce99496c81039"}
Jul 09 18:05:14.668241 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:14.668193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz" event={"ID":"1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e","Type":"ContainerStarted","Data":"5cd80a79c2b3fbaa9468a04d8242f2931bfc47e4894f03b3374bfb1ca2b5d58d"}
Jul 09 18:05:14.689179 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:14.688035    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9" event={"ID":"603013ff-71ad-4016-b7cb-8bdf4f65448d","Type":"ContainerStarted","Data":"31324315c5f10780ae46467e615d9f822ecdc572b3f7f4e5e8d1f15407ed076c"}
Jul 09 18:05:15.569169 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:15.569125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:15.569364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:15.569202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:15.569364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:15.569250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:15.569364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:15.569282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:15.569520 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:15.569504    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:05:15.569588 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:15.569577    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs podName:c5c938ef-4491-4fc7-aacb-02fa8e3ce61b nodeName:}" failed. No retries permitted until 2026-07-09 18:05:19.569556314 +0000 UTC m=+42.756534102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs") pod "router-default-675d564c74-s5fsx" (UID: "c5c938ef-4491-4fc7-aacb-02fa8e3ce61b") : secret "router-metrics-certs-default" not found
Jul 09 18:05:15.570079 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:15.570059    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle podName:c5c938ef-4491-4fc7-aacb-02fa8e3ce61b nodeName:}" failed. No retries permitted until 2026-07-09 18:05:19.570040422 +0000 UTC m=+42.757018183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle") pod "router-default-675d564c74-s5fsx" (UID: "c5c938ef-4491-4fc7-aacb-02fa8e3ce61b") : configmap references non-existent config key: service-ca.crt
Jul 09 18:05:15.570193 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:15.570141    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:05:15.570193 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:15.570186    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls podName:1d25c758-d9bb-4113-a71d-de846c62e5ef nodeName:}" failed. No retries permitted until 2026-07-09 18:05:19.570172906 +0000 UTC m=+42.757150653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-cjt5m" (UID: "1d25c758-d9bb-4113-a71d-de846c62e5ef") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:05:15.570311 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:15.570251    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:05:15.570311 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:15.570262    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6846c96d87-7mhdw: secret "image-registry-tls" not found
Jul 09 18:05:15.570311 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:15.570293    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls podName:f6363dd9-59ec-4cbd-ae82-a9e896a0814c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:19.570282018 +0000 UTC m=+42.757259777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls") pod "image-registry-6846c96d87-7mhdw" (UID: "f6363dd9-59ec-4cbd-ae82-a9e896a0814c") : secret "image-registry-tls" not found
Jul 09 18:05:15.670827 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:15.670769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2wnng\" (UID: \"4b9b442e-e7e5-4b54-bd16-5141660c77ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"
Jul 09 18:05:15.670827 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:15.670843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:15.671516 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:15.671007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9zf8g\" (UID: \"fccda6c1-c857-41be-a4fd-198cbd82a2c5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:05:15.671516 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:15.671162    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:05:15.671516 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:15.671236    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert podName:fccda6c1-c857-41be-a4fd-198cbd82a2c5 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:19.671216509 +0000 UTC m=+42.858194270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-9zf8g" (UID: "fccda6c1-c857-41be-a4fd-198cbd82a2c5") : secret "networking-console-plugin-cert" not found
Jul 09 18:05:15.672140 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:15.671769    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:05:15.672140 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:15.671847    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls podName:4b9b442e-e7e5-4b54-bd16-5141660c77ce nodeName:}" failed. No retries permitted until 2026-07-09 18:05:19.671829439 +0000 UTC m=+42.858807190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls") pod "cluster-samples-operator-65db858894-2wnng" (UID: "4b9b442e-e7e5-4b54-bd16-5141660c77ce") : secret "samples-operator-tls" not found
Jul 09 18:05:15.672140 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:15.671908    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:15.672140 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:15.671938    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls podName:122683df-b7fe-4c5a-9974-987713ba1ee2 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:19.671928269 +0000 UTC m=+42.858906020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls") pod "dns-default-p7zkv" (UID: "122683df-b7fe-4c5a-9974-987713ba1ee2") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:16.730172 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:16.730078    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pf6x5" event={"ID":"7bb277a6-db3b-46bb-b1d1-ffca6119ec59","Type":"ContainerStarted","Data":"e27709a6962227b1268bed39c5752d64b67e20a874c1e140bad93900598264b3"}
Jul 09 18:05:16.758687 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:16.757899    2578 generic.go:356] "Generic (PLEG): container finished" podID="d9f223b1-0765-46bb-b8af-136743836d8a" containerID="5043550e5ca01d241c4e0608255c8fdc474579e51883b148b64cc09a82223251" exitCode=0
Jul 09 18:05:16.758687 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:16.757976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jmw8" event={"ID":"d9f223b1-0765-46bb-b8af-136743836d8a","Type":"ContainerDied","Data":"5043550e5ca01d241c4e0608255c8fdc474579e51883b148b64cc09a82223251"}
Jul 09 18:05:16.815428 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:16.812044    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pf6x5" podStartSLOduration=20.249460766 podStartE2EDuration="22.812019946s" totalImagesPullingTime="2.56255918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:16.775519006 +0000 UTC m=+39.962496800" watchObservedRunningTime="2026-07-09 18:05:16.812019946 +0000 UTC m=+39.998997715"
Jul 09 18:05:16.980533 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:16.980433    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.723287 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:17.723251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.829857 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:17.829814    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:19.298844 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:19.298811    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:19.431034 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:19.431001    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:19.621700 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:19.621607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:19.621700 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:19.621667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:19.621700 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:19.621702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:19.622002 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:19.621791    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:05:19.622002 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:19.621873    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls podName:1d25c758-d9bb-4113-a71d-de846c62e5ef nodeName:}" failed. No retries permitted until 2026-07-09 18:05:27.621854237 +0000 UTC m=+50.808831983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-cjt5m" (UID: "1d25c758-d9bb-4113-a71d-de846c62e5ef") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:05:19.622002 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:19.621878    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:05:19.622002 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:19.621899    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6846c96d87-7mhdw: secret "image-registry-tls" not found
Jul 09 18:05:19.622002 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:19.621900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:19.622002 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:19.621919    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:05:19.622002 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:19.621955    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls podName:f6363dd9-59ec-4cbd-ae82-a9e896a0814c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:27.621936591 +0000 UTC m=+50.808914337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls") pod "image-registry-6846c96d87-7mhdw" (UID: "f6363dd9-59ec-4cbd-ae82-a9e896a0814c") : secret "image-registry-tls" not found
Jul 09 18:05:19.622002 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:19.621975    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs podName:c5c938ef-4491-4fc7-aacb-02fa8e3ce61b nodeName:}" failed. No retries permitted until 2026-07-09 18:05:27.621965307 +0000 UTC m=+50.808943058 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs") pod "router-default-675d564c74-s5fsx" (UID: "c5c938ef-4491-4fc7-aacb-02fa8e3ce61b") : secret "router-metrics-certs-default" not found
Jul 09 18:05:19.622361 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:19.622010    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle podName:c5c938ef-4491-4fc7-aacb-02fa8e3ce61b nodeName:}" failed. No retries permitted until 2026-07-09 18:05:27.621999792 +0000 UTC m=+50.808977538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle") pod "router-default-675d564c74-s5fsx" (UID: "c5c938ef-4491-4fc7-aacb-02fa8e3ce61b") : configmap references non-existent config key: service-ca.crt
Jul 09 18:05:19.722918 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:19.722881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9zf8g\" (UID: \"fccda6c1-c857-41be-a4fd-198cbd82a2c5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:05:19.723088 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:19.722970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2wnng\" (UID: \"4b9b442e-e7e5-4b54-bd16-5141660c77ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"
Jul 09 18:05:19.723088 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:19.722999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:19.723088 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:19.723076    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:05:19.723257 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:19.723134    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:05:19.723257 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:19.723139    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert podName:fccda6c1-c857-41be-a4fd-198cbd82a2c5 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:27.723121047 +0000 UTC m=+50.910098793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-9zf8g" (UID: "fccda6c1-c857-41be-a4fd-198cbd82a2c5") : secret "networking-console-plugin-cert" not found
Jul 09 18:05:19.723257 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:19.723183    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:19.723257 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:19.723201    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls podName:4b9b442e-e7e5-4b54-bd16-5141660c77ce nodeName:}" failed. No retries permitted until 2026-07-09 18:05:27.72318338 +0000 UTC m=+50.910161128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls") pod "cluster-samples-operator-65db858894-2wnng" (UID: "4b9b442e-e7e5-4b54-bd16-5141660c77ce") : secret "samples-operator-tls" not found
Jul 09 18:05:19.723257 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:19.723226    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls podName:122683df-b7fe-4c5a-9974-987713ba1ee2 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:27.723212397 +0000 UTC m=+50.910190170 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls") pod "dns-default-p7zkv" (UID: "122683df-b7fe-4c5a-9974-987713ba1ee2") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:20.140534 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:20.140499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:20.243119 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:20.243078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:20.353008 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:20.352969    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:20.452813 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:20.452770    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:20.554588 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:20.554547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:20.637571 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:20.637537    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:20.716457 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:20.716378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:21.779626 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:21.779584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jmw8" event={"ID":"d9f223b1-0765-46bb-b8af-136743836d8a","Type":"ContainerStarted","Data":"42d6b0adada6216cb9917df2f0c67e828d3ea66f157a6ea5bb9c3119ae758203"}
Jul 09 18:05:21.821127 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:21.821069    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8jmw8" podStartSLOduration=17.766527119 podStartE2EDuration="44.821046767s" totalImagesPullingTime="27.054519648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:21.820046426 +0000 UTC m=+45.007024193" watchObservedRunningTime="2026-07-09 18:05:21.821046767 +0000 UTC m=+45.008024537"
Jul 09 18:05:22.022843 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:22.022788    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:22.154077 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:22.153982    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:23.809591 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:23.809516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:27.702270 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.702170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:27.702270 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.702237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:27.702854 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.702280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:27.702854 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.702313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:27.702854 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:27.702513    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:05:27.702854 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:27.702572    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs podName:c5c938ef-4491-4fc7-aacb-02fa8e3ce61b nodeName:}" failed. No retries permitted until 2026-07-09 18:05:43.702553364 +0000 UTC m=+66.889531118 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs") pod "router-default-675d564c74-s5fsx" (UID: "c5c938ef-4491-4fc7-aacb-02fa8e3ce61b") : secret "router-metrics-certs-default" not found
Jul 09 18:05:27.702854 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:27.702660    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle podName:c5c938ef-4491-4fc7-aacb-02fa8e3ce61b nodeName:}" failed. No retries permitted until 2026-07-09 18:05:43.702649579 +0000 UTC m=+66.889627330 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle") pod "router-default-675d564c74-s5fsx" (UID: "c5c938ef-4491-4fc7-aacb-02fa8e3ce61b") : configmap references non-existent config key: service-ca.crt
Jul 09 18:05:27.702854 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:27.702784    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:05:27.702854 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:27.702821    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6846c96d87-7mhdw: secret "image-registry-tls" not found
Jul 09 18:05:27.702854 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:27.702838    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:05:27.703226 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:27.702873    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls podName:f6363dd9-59ec-4cbd-ae82-a9e896a0814c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:43.702856327 +0000 UTC m=+66.889834089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls") pod "image-registry-6846c96d87-7mhdw" (UID: "f6363dd9-59ec-4cbd-ae82-a9e896a0814c") : secret "image-registry-tls" not found
Jul 09 18:05:27.703226 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:27.702898    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls podName:1d25c758-d9bb-4113-a71d-de846c62e5ef nodeName:}" failed. No retries permitted until 2026-07-09 18:05:43.70288329 +0000 UTC m=+66.889861037 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-cjt5m" (UID: "1d25c758-d9bb-4113-a71d-de846c62e5ef") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:05:27.804948 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.804916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9zf8g\" (UID: \"fccda6c1-c857-41be-a4fd-198cbd82a2c5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:05:27.805238 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.805221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2wnng\" (UID: \"4b9b442e-e7e5-4b54-bd16-5141660c77ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"
Jul 09 18:05:27.805340 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.805328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:27.805583 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:27.805063    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:05:27.805713 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:27.805704    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert podName:fccda6c1-c857-41be-a4fd-198cbd82a2c5 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:43.805684877 +0000 UTC m=+66.992662629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-9zf8g" (UID: "fccda6c1-c857-41be-a4fd-198cbd82a2c5") : secret "networking-console-plugin-cert" not found
Jul 09 18:05:27.805822 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:27.805570    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:27.805944 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:27.805935    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls podName:122683df-b7fe-4c5a-9974-987713ba1ee2 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:43.805920556 +0000 UTC m=+66.992898306 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls") pod "dns-default-p7zkv" (UID: "122683df-b7fe-4c5a-9974-987713ba1ee2") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:27.806201 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:27.805849    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:05:27.806323 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:27.806315    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls podName:4b9b442e-e7e5-4b54-bd16-5141660c77ce nodeName:}" failed. No retries permitted until 2026-07-09 18:05:43.806302267 +0000 UTC m=+66.993280021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls") pod "cluster-samples-operator-65db858894-2wnng" (UID: "4b9b442e-e7e5-4b54-bd16-5141660c77ce") : secret "samples-operator-tls" not found
Jul 09 18:05:27.808179 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.808039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzjcd" event={"ID":"3b4a856f-2c4e-45b0-8d6f-25cce65d42a0","Type":"ContainerStarted","Data":"f6dbad314ab71f3be48efa73b6f95a2904a108b9c510e0b022e0a1bb66858660"}
Jul 09 18:05:27.810086 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.810003    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz" event={"ID":"1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e","Type":"ContainerStarted","Data":"788c857370fea05ae4aee3e378036dcf18f8e819bba801caa7ca0a1ce0ed39c3"}
Jul 09 18:05:27.818322 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.818288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9" event={"ID":"603013ff-71ad-4016-b7cb-8bdf4f65448d","Type":"ContainerStarted","Data":"4970fa5574fb17d56fecdf6e1adc1c798f0c9fb1087dfaf5067b28d4f2e51bc0"}
Jul 09 18:05:27.819098 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.819077    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"
Jul 09 18:05:27.820584 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.820548    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k" event={"ID":"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b","Type":"ContainerStarted","Data":"cc37c16997233e28d4185f05ad8e988fa03358ea086a2434a87ecd05eb2bd099"}
Jul 09 18:05:27.821430 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.821250    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9"
Jul 09 18:05:27.823112 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.823088    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b45b454c8-r66xm" event={"ID":"924ac8cc-82c8-40e7-995d-e10eb77aa558","Type":"ContainerStarted","Data":"f6aff779533f1a838aef0156e6cae30f92c2c6612c9bce814f044146b8d246f0"}
Jul 09 18:05:27.826564 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.826537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw" event={"ID":"e815768d-07d5-486e-81c3-d29a321181a0","Type":"ContainerStarted","Data":"ec2ad9a37110a512758e4bfc6fdddf497c5af9de706f7f391251575e94e0e745"}
Jul 09 18:05:27.829104 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.829075    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/0.log"
Jul 09 18:05:27.829218 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.829160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-kln87" event={"ID":"806524a3-c51c-49e3-822e-c04ed12d98cd","Type":"ContainerStarted","Data":"df7c1139b029a02dc48ccd59e87ad40483565d993eac960407c852ba47b805de"}
Jul 09 18:05:27.829483 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.829467    2578 scope.go:117] "RemoveContainer" containerID="df7c1139b029a02dc48ccd59e87ad40483565d993eac960407c852ba47b805de"
Jul 09 18:05:27.834963 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.834870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6" event={"ID":"12137a00-1ea6-420d-84be-97efa63bffe8","Type":"ContainerStarted","Data":"a01774f8a02b23d208279e08993e618447d40c5ac86edbeef9e948dc39531532"}
Jul 09 18:05:27.834963 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.834935    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6" event={"ID":"12137a00-1ea6-420d-84be-97efa63bffe8","Type":"ContainerStarted","Data":"f25662faccf8f1094267de538f87f5049866ec700fc5ee0dd739e72e0484785b"}
Jul 09 18:05:27.836787 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.836763    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b56mt" event={"ID":"2c211d21-6c7a-4c79-a433-751b85d263b4","Type":"ContainerStarted","Data":"05bd2cb24ff8c166d9212dda96c7e58f6c3e6ce6e9103c896367abca8c0b6fcd"}
Jul 09 18:05:27.852487 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.852431    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzjcd" podStartSLOduration=16.138004183 podStartE2EDuration="29.852410516s" totalImagesPullingTime="13.714406333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:27.850527859 +0000 UTC m=+51.037505651" watchObservedRunningTime="2026-07-09 18:05:27.852410516 +0000 UTC m=+51.039388287"
Jul 09 18:05:27.855268 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.855230    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:27.929953 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.929889    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c5d99496-vl2z9" podStartSLOduration=36.123806657 podStartE2EDuration="49.929871947s" totalImagesPullingTime="13.80606529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:27.927569554 +0000 UTC m=+51.114547322" watchObservedRunningTime="2026-07-09 18:05:27.929871947 +0000 UTC m=+51.116849714"
Jul 09 18:05:27.930821 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:27.930771    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz" podStartSLOduration=20.093835416 podStartE2EDuration="33.930761279s" totalImagesPullingTime="13.836925863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:27.894995704 +0000 UTC m=+51.081973496" watchObservedRunningTime="2026-07-09 18:05:27.930761279 +0000 UTC m=+51.117739066"
Jul 09 18:05:28.000441 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.000265    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b45b454c8-r66xm" podStartSLOduration=36.286663456 podStartE2EDuration="50.000243125s" totalImagesPullingTime="13.713579669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:27.988655999 +0000 UTC m=+51.175633798" watchObservedRunningTime="2026-07-09 18:05:28.000243125 +0000 UTC m=+51.187220918"
Jul 09 18:05:28.022015 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.021979    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.097220 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.097158    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw" podStartSLOduration=20.373941708 podStartE2EDuration="34.097135146s" totalImagesPullingTime="13.723193438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:28.096222547 +0000 UTC m=+51.283200317" watchObservedRunningTime="2026-07-09 18:05:28.097135146 +0000 UTC m=+51.284112914"
Jul 09 18:05:28.150426 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.150215    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b56mt" podStartSLOduration=3.34628842 podStartE2EDuration="17.15018999s" totalImagesPullingTime="13.80390157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:28.14845554 +0000 UTC m=+51.335433311" watchObservedRunningTime="2026-07-09 18:05:28.15018999 +0000 UTC m=+51.337167760"
Jul 09 18:05:28.215478 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.214733    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k" podStartSLOduration=20.498312209 podStartE2EDuration="34.214708318s" totalImagesPullingTime="13.716396109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:28.213346493 +0000 UTC m=+51.400324262" watchObservedRunningTime="2026-07-09 18:05:28.214708318 +0000 UTC m=+51.401686086"
Jul 09 18:05:28.843616 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.843584    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:05:28.844107 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.844088    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/0.log"
Jul 09 18:05:28.844183 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.844123    2578 generic.go:356] "Generic (PLEG): container finished" podID="806524a3-c51c-49e3-822e-c04ed12d98cd" containerID="df7c1139b029a02dc48ccd59e87ad40483565d993eac960407c852ba47b805de" exitCode=255
Jul 09 18:05:28.844183 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.844141    2578 generic.go:356] "Generic (PLEG): container finished" podID="806524a3-c51c-49e3-822e-c04ed12d98cd" containerID="c849e623af70bc007ab3e821014760a8e51b1f0b1507452de76ea3c18f5a4fed" exitCode=255
Jul 09 18:05:28.844320 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.844208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-kln87" event={"ID":"806524a3-c51c-49e3-822e-c04ed12d98cd","Type":"ContainerDied","Data":"df7c1139b029a02dc48ccd59e87ad40483565d993eac960407c852ba47b805de"}
Jul 09 18:05:28.844320 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.844294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-kln87" event={"ID":"806524a3-c51c-49e3-822e-c04ed12d98cd","Type":"ContainerDied","Data":"c849e623af70bc007ab3e821014760a8e51b1f0b1507452de76ea3c18f5a4fed"}
Jul 09 18:05:28.844416 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.844321    2578 scope.go:117] "RemoveContainer" containerID="df7c1139b029a02dc48ccd59e87ad40483565d993eac960407c852ba47b805de"
Jul 09 18:05:28.844748 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.844723    2578 scope.go:117] "RemoveContainer" containerID="c849e623af70bc007ab3e821014760a8e51b1f0b1507452de76ea3c18f5a4fed"
Jul 09 18:05:28.845203 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:28.844942    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-kln87_openshift-console-operator(806524a3-c51c-49e3-822e-c04ed12d98cd)\"" pod="openshift-console-operator/console-operator-7866497494-kln87" podUID="806524a3-c51c-49e3-822e-c04ed12d98cd"
Jul 09 18:05:28.847610 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.847580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6" event={"ID":"12137a00-1ea6-420d-84be-97efa63bffe8","Type":"ContainerStarted","Data":"70f80621f49c4c66e7109bb30f9407029e76784cf0b20a0801ae5cd27d934fa9"}
Jul 09 18:05:28.887222 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.887197    2578 scope.go:117] "RemoveContainer" containerID="df7c1139b029a02dc48ccd59e87ad40483565d993eac960407c852ba47b805de"
Jul 09 18:05:28.887608 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:28.887585    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7c1139b029a02dc48ccd59e87ad40483565d993eac960407c852ba47b805de\": container with ID starting with df7c1139b029a02dc48ccd59e87ad40483565d993eac960407c852ba47b805de not found: ID does not exist" containerID="df7c1139b029a02dc48ccd59e87ad40483565d993eac960407c852ba47b805de"
Jul 09 18:05:28.887706 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.887621    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7c1139b029a02dc48ccd59e87ad40483565d993eac960407c852ba47b805de"} err="failed to get container status \"df7c1139b029a02dc48ccd59e87ad40483565d993eac960407c852ba47b805de\": rpc error: code = NotFound desc = could not find container \"df7c1139b029a02dc48ccd59e87ad40483565d993eac960407c852ba47b805de\": container with ID starting with df7c1139b029a02dc48ccd59e87ad40483565d993eac960407c852ba47b805de not found: ID does not exist"
Jul 09 18:05:28.890476 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:28.890403    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6" podStartSLOduration=37.178397759 podStartE2EDuration="50.890381749s" totalImagesPullingTime="13.71198399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:28.889810708 +0000 UTC m=+52.076788477" watchObservedRunningTime="2026-07-09 18:05:28.890381749 +0000 UTC m=+52.077359510"
Jul 09 18:05:29.488005 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.487965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:29.586095 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.586058    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dnhg8"]
Jul 09 18:05:29.618531 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.618476    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dnhg8"]
Jul 09 18:05:29.618713 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.618629    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:29.621868 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.621838    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 18:05:29.621868 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.621843    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qh478\""
Jul 09 18:05:29.621868 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.621863    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 18:05:29.622119 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.622075    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:29.734030 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.733986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-data-volume\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:29.734214 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.734194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:29.734260 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.734233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fzhc\" (UniqueName: \"kubernetes.io/projected/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-api-access-2fzhc\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:29.734320 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.734264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:29.734405 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.734384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-crio-socket\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:29.763699 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.763611    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:29.835683 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.835635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:29.835919 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.835694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fzhc\" (UniqueName: \"kubernetes.io/projected/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-api-access-2fzhc\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:29.835919 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.835728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:29.835919 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:29.835786    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:29.835919 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:29.835878    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:05:29.836090 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:29.835885    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls podName:f7b733c6-d68b-4be5-af36-4c7f7d8cddf0 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:30.335865407 +0000 UTC m=+53.522843153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnhg8" (UID: "f7b733c6-d68b-4be5-af36-4c7f7d8cddf0") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:29.836090 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.835953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-crio-socket\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:29.836090 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:29.835975    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm podName:f7b733c6-d68b-4be5-af36-4c7f7d8cddf0 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:30.335958456 +0000 UTC m=+53.522936208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnhg8" (UID: "f7b733c6-d68b-4be5-af36-4c7f7d8cddf0") : configmap "kube-rbac-proxy" not found
Jul 09 18:05:29.836090 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.836021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-data-volume\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:29.836090 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.836023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-crio-socket\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:29.836417 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.836399    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-data-volume\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:29.851220 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.851178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fzhc\" (UniqueName: \"kubernetes.io/projected/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-api-access-2fzhc\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:29.852780 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.852761    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:05:29.853229 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:29.853209    2578 scope.go:117] "RemoveContainer" containerID="c849e623af70bc007ab3e821014760a8e51b1f0b1507452de76ea3c18f5a4fed"
Jul 09 18:05:29.853438 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:29.853413    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-kln87_openshift-console-operator(806524a3-c51c-49e3-822e-c04ed12d98cd)\"" pod="openshift-console-operator/console-operator-7866497494-kln87" podUID="806524a3-c51c-49e3-822e-c04ed12d98cd"
Jul 09 18:05:30.106529 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:30.106443    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:30.195147 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:30.195113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:30.341768 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:30.341730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:30.341972 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:30.341787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:30.341972 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:30.341879    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:05:30.341972 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:30.341902    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:30.341972 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:30.341951    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm podName:f7b733c6-d68b-4be5-af36-4c7f7d8cddf0 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:31.341935432 +0000 UTC m=+54.528913178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnhg8" (UID: "f7b733c6-d68b-4be5-af36-4c7f7d8cddf0") : configmap "kube-rbac-proxy" not found
Jul 09 18:05:30.341972 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:30.341970    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls podName:f7b733c6-d68b-4be5-af36-4c7f7d8cddf0 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:31.341961765 +0000 UTC m=+54.528939511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnhg8" (UID: "f7b733c6-d68b-4be5-af36-4c7f7d8cddf0") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:30.767051 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:30.767017    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:30.871219 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:30.871179    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:31.350869 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:31.350827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:31.351087 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:31.350883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:31.351087 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:31.350974    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:31.351087 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:31.351004    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:05:31.351087 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:31.351048    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls podName:f7b733c6-d68b-4be5-af36-4c7f7d8cddf0 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:33.351029789 +0000 UTC m=+56.538007547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnhg8" (UID: "f7b733c6-d68b-4be5-af36-4c7f7d8cddf0") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:31.351292 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:31.351091    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm podName:f7b733c6-d68b-4be5-af36-4c7f7d8cddf0 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:33.351076194 +0000 UTC m=+56.538053940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnhg8" (UID: "f7b733c6-d68b-4be5-af36-4c7f7d8cddf0") : configmap "kube-rbac-proxy" not found
Jul 09 18:05:32.094622 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:32.094566    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:32.095045 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:32.094652    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:32.095045 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:32.095028    2578 scope.go:117] "RemoveContainer" containerID="c849e623af70bc007ab3e821014760a8e51b1f0b1507452de76ea3c18f5a4fed"
Jul 09 18:05:32.095200 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:32.095180    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-kln87_openshift-console-operator(806524a3-c51c-49e3-822e-c04ed12d98cd)\"" pod="openshift-console-operator/console-operator-7866497494-kln87" podUID="806524a3-c51c-49e3-822e-c04ed12d98cd"
Jul 09 18:05:32.280361 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:32.280321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:32.420722 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:32.420636    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:32.540276 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:32.540240    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:32.572610 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:32.572581    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d7pqh_9ccfb877-5ee9-40fa-a28d-8310e8922463/dns-node-resolver/0.log"
Jul 09 18:05:33.370588 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:33.370547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:33.371096 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:33.370600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:33.371096 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:33.370701    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:33.371096 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:33.370754    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:05:33.371096 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:33.370816    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls podName:f7b733c6-d68b-4be5-af36-4c7f7d8cddf0 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:37.370772934 +0000 UTC m=+60.557750725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnhg8" (UID: "f7b733c6-d68b-4be5-af36-4c7f7d8cddf0") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:33.371096 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:33.370837    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm podName:f7b733c6-d68b-4be5-af36-4c7f7d8cddf0 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:37.370828723 +0000 UTC m=+60.557806474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnhg8" (UID: "f7b733c6-d68b-4be5-af36-4c7f7d8cddf0") : configmap "kube-rbac-proxy" not found
Jul 09 18:05:33.373342 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:33.373318    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-klwxw_c290aebc-bdd2-47e7-a307-f00b1df88b5f/node-ca/0.log"
Jul 09 18:05:33.613377 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:33.613339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:33.916346 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:33.916289    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:34.173875 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:34.173764    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b56mt_2c211d21-6c7a-4c79-a433-751b85d263b4/serve-healthcheck-canary/0.log"
Jul 09 18:05:34.973506 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:34.973465    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-54gvz_1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e/kube-storage-version-migrator-operator/0.log"
Jul 09 18:05:36.551205 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:36.551168    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlk4c"
Jul 09 18:05:37.361055 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:37.361018    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:37.408844 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:37.408786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:37.409020 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:37.408863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:37.409020 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:37.408902    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:37.409020 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:37.408969    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls podName:f7b733c6-d68b-4be5-af36-4c7f7d8cddf0 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:45.408953997 +0000 UTC m=+68.595931743 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnhg8" (UID: "f7b733c6-d68b-4be5-af36-4c7f7d8cddf0") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:37.409177 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:37.408967    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:05:37.409177 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:37.409111    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm podName:f7b733c6-d68b-4be5-af36-4c7f7d8cddf0 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:45.409089151 +0000 UTC m=+68.596066917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnhg8" (UID: "f7b733c6-d68b-4be5-af36-4c7f7d8cddf0") : configmap "kube-rbac-proxy" not found
Jul 09 18:05:37.486473 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:37.486441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:37.602045 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:37.602008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:37.704115 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:37.704084    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.292558 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:38.292527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.649039 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:38.648948    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.997646 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:38.997613    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:39.444688 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:39.444606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:39.920226 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:39.920189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:40.037103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:40.037071    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:40.236435 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:40.236402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:40.638108 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:40.638032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:41.205315 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:41.205279    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:41.309473 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:41.309439    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:41.682645 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:41.682567    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:41.839470 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:41.839440    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:42.442240 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:42.442206    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:42.853153 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:42.853122    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.052883 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.052846    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.163090 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.163002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62gd\" (UniqueName: \"kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd\") pod \"network-check-target-f6494\" (UID: \"62eb9ea7-9552-4c47-a202-adbba7cd45fa\") " pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:05:43.163090 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.163056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs\") pod \"network-metrics-daemon-kpcpn\" (UID: \"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58\") " pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:05:43.163653 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.163626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:05:43.168406 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.168326    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:43.168704 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.168669    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:43.169297 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.169273    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j62gd\" (UniqueName: \"kubernetes.io/projected/62eb9ea7-9552-4c47-a202-adbba7cd45fa-kube-api-access-j62gd\") pod \"network-check-target-f6494\" (UID: \"62eb9ea7-9552-4c47-a202-adbba7cd45fa\") " pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:05:43.174659 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:43.174633    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 18:05:43.174819 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:43.174709    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs podName:11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58 nodeName:}" failed. No retries permitted until 2026-07-09 18:06:47.174687125 +0000 UTC m=+130.361664884 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs") pod "network-metrics-daemon-kpcpn" (UID: "11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58") : secret "metrics-daemon-secret" not found
Jul 09 18:05:43.176858 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.176830    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3da572d-acbd-4c3c-a693-c0eac330f719-original-pull-secret\") pod \"global-pull-secret-syncer-5kvp7\" (UID: \"d3da572d-acbd-4c3c-a693-c0eac330f719\") " pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:05:43.317669 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.317632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5kvp7"
Jul 09 18:05:43.334007 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.333977    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vcghf\""
Jul 09 18:05:43.341742 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.341710    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:05:43.442781 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.442731    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.458397 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.458322    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5kvp7"]
Jul 09 18:05:43.461906 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:05:43.461871    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3da572d_acbd_4c3c_a693_c0eac330f719.slice/crio-7e7f684854fc921fe45358dff1f1b775dcc8007a9593f03ad690ba3d6c6ef188 WatchSource:0}: Error finding container 7e7f684854fc921fe45358dff1f1b775dcc8007a9593f03ad690ba3d6c6ef188: Status 404 returned error can't find the container with id 7e7f684854fc921fe45358dff1f1b775dcc8007a9593f03ad690ba3d6c6ef188
Jul 09 18:05:43.491228 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.491198    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f6494"]
Jul 09 18:05:43.493672 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:05:43.493642    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62eb9ea7_9552_4c47_a202_adbba7cd45fa.slice/crio-cd14f8df4ef2aa553dfe8a6c5f5e0ceec7859f46531d6864b0600a2f5eb258d1 WatchSource:0}: Error finding container cd14f8df4ef2aa553dfe8a6c5f5e0ceec7859f46531d6864b0600a2f5eb258d1: Status 404 returned error can't find the container with id cd14f8df4ef2aa553dfe8a6c5f5e0ceec7859f46531d6864b0600a2f5eb258d1
Jul 09 18:05:43.626768 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.626728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.769978 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.769939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:43.770166 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.769987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:05:43.770166 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.770025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:43.770166 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.770047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:43.770166 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:43.770159    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:05:43.770387 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:43.770231    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls podName:1d25c758-d9bb-4113-a71d-de846c62e5ef nodeName:}" failed. No retries permitted until 2026-07-09 18:06:15.770211603 +0000 UTC m=+98.957189361 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-cjt5m" (UID: "1d25c758-d9bb-4113-a71d-de846c62e5ef") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:05:43.770613 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.770586    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-service-ca-bundle\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:43.772461 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.772439    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls\") pod \"image-registry-6846c96d87-7mhdw\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") " pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:43.772551 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.772463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c938ef-4491-4fc7-aacb-02fa8e3ce61b-metrics-certs\") pod \"router-default-675d564c74-s5fsx\" (UID: \"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b\") " pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:43.828090 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.828035    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.868506 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.868479    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rfwbw\""
Jul 09 18:05:43.870836 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.870788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9zf8g\" (UID: \"fccda6c1-c857-41be-a4fd-198cbd82a2c5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:05:43.870921 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.870892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2wnng\" (UID: \"4b9b442e-e7e5-4b54-bd16-5141660c77ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"
Jul 09 18:05:43.870921 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.870917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:43.870997 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:43.870953    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:05:43.871055 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:43.871044    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert podName:fccda6c1-c857-41be-a4fd-198cbd82a2c5 nodeName:}" failed. No retries permitted until 2026-07-09 18:06:15.871027059 +0000 UTC m=+99.058004805 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-9zf8g" (UID: "fccda6c1-c857-41be-a4fd-198cbd82a2c5") : secret "networking-console-plugin-cert" not found
Jul 09 18:05:43.873238 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.873214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/122683df-b7fe-4c5a-9974-987713ba1ee2-metrics-tls\") pod \"dns-default-p7zkv\" (UID: \"122683df-b7fe-4c5a-9974-987713ba1ee2\") " pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:43.873368 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.873348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b9b442e-e7e5-4b54-bd16-5141660c77ce-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2wnng\" (UID: \"4b9b442e-e7e5-4b54-bd16-5141660c77ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"
Jul 09 18:05:43.876286 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.876252    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:43.889891 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.889859    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4mr58\""
Jul 09 18:05:43.897363 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.897329    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:43.903739 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.903704    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5kvp7" event={"ID":"d3da572d-acbd-4c3c-a693-c0eac330f719","Type":"ContainerStarted","Data":"7e7f684854fc921fe45358dff1f1b775dcc8007a9593f03ad690ba3d6c6ef188"}
Jul 09 18:05:43.905373 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.905344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f6494" event={"ID":"62eb9ea7-9552-4c47-a202-adbba7cd45fa","Type":"ContainerStarted","Data":"afefe3f1945e2a2114dc6ee90eb69cc2eb312243833be9023f3f8843de5a7adc"}
Jul 09 18:05:43.905501 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.905388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f6494" event={"ID":"62eb9ea7-9552-4c47-a202-adbba7cd45fa","Type":"ContainerStarted","Data":"cd14f8df4ef2aa553dfe8a6c5f5e0ceec7859f46531d6864b0600a2f5eb258d1"}
Jul 09 18:05:43.905501 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.905480    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:05:43.932445 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.932362    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f6494" podStartSLOduration=66.932305582 podStartE2EDuration="1m6.932305582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:43.931313383 +0000 UTC m=+67.118291151" watchObservedRunningTime="2026-07-09 18:05:43.932305582 +0000 UTC m=+67.119283351"
Jul 09 18:05:43.945606 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.945576    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nq7m5\""
Jul 09 18:05:43.953432 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.953354    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:43.955117 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.955039    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gqm28\""
Jul 09 18:05:43.966058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:43.965706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"
Jul 09 18:05:44.028348 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.028244    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-675d564c74-s5fsx"]
Jul 09 18:05:44.040684 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:05:44.040629    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c938ef_4491_4fc7_aacb_02fa8e3ce61b.slice/crio-73737bba5848757cf6f2797fda1761834800d5b96a23328be12ec2096c280286 WatchSource:0}: Error finding container 73737bba5848757cf6f2797fda1761834800d5b96a23328be12ec2096c280286: Status 404 returned error can't find the container with id 73737bba5848757cf6f2797fda1761834800d5b96a23328be12ec2096c280286
Jul 09 18:05:44.069016 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.068984    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6846c96d87-7mhdw"]
Jul 09 18:05:44.088704 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:05:44.088673    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6363dd9_59ec_4cbd_ae82_a9e896a0814c.slice/crio-4774dbeb8ef94a9471242f8379c6b1141d5f66c682c82c013d7d0f6f8098295c WatchSource:0}: Error finding container 4774dbeb8ef94a9471242f8379c6b1141d5f66c682c82c013d7d0f6f8098295c: Status 404 returned error can't find the container with id 4774dbeb8ef94a9471242f8379c6b1141d5f66c682c82c013d7d0f6f8098295c
Jul 09 18:05:44.127437 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.127357    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p7zkv"]
Jul 09 18:05:44.137958 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:05:44.137917    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122683df_b7fe_4c5a_9974_987713ba1ee2.slice/crio-e25bb64217e97cbe3f6b4414a14fb985fb4a2f5876d5c040cc04cfb9726cbf91 WatchSource:0}: Error finding container e25bb64217e97cbe3f6b4414a14fb985fb4a2f5876d5c040cc04cfb9726cbf91: Status 404 returned error can't find the container with id e25bb64217e97cbe3f6b4414a14fb985fb4a2f5876d5c040cc04cfb9726cbf91
Jul 09 18:05:44.147016 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.146987    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng"]
Jul 09 18:05:44.259028 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.258973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:44.397771 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.397688    2578 scope.go:117] "RemoveContainer" containerID="c849e623af70bc007ab3e821014760a8e51b1f0b1507452de76ea3c18f5a4fed"
Jul 09 18:05:44.642142 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.642103    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:44.860069 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.860031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:44.914942 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.914901    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng" event={"ID":"4b9b442e-e7e5-4b54-bd16-5141660c77ce","Type":"ContainerStarted","Data":"acfd0f12b4fdc74b894b00610466149e0a2d6060138900a66f3f35c30d3cae74"}
Jul 09 18:05:44.917050 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.916971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p7zkv" event={"ID":"122683df-b7fe-4c5a-9974-987713ba1ee2","Type":"ContainerStarted","Data":"e25bb64217e97cbe3f6b4414a14fb985fb4a2f5876d5c040cc04cfb9726cbf91"}
Jul 09 18:05:44.918843 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.918790    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-675d564c74-s5fsx" event={"ID":"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b","Type":"ContainerStarted","Data":"5864bae26ee3934d219dccd507819d8f1adfca6f5026094ca54fe044d7c8984f"}
Jul 09 18:05:44.918971 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.918867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-675d564c74-s5fsx" event={"ID":"c5c938ef-4491-4fc7-aacb-02fa8e3ce61b","Type":"ContainerStarted","Data":"73737bba5848757cf6f2797fda1761834800d5b96a23328be12ec2096c280286"}
Jul 09 18:05:44.921768 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.921747    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:05:44.921916 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.921828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-kln87" event={"ID":"806524a3-c51c-49e3-822e-c04ed12d98cd","Type":"ContainerStarted","Data":"099870ee11cb766e5bd3b229de47d9c11f38b0f5d10e75c97794ff3741db00ea"}
Jul 09 18:05:44.922519 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.922366    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:44.924034 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.924009    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6846c96d87-7mhdw" event={"ID":"f6363dd9-59ec-4cbd-ae82-a9e896a0814c","Type":"ContainerStarted","Data":"0bdc4b370f9b30866d6cbd2511d6ddad96a02f8928e4b7694eaea58cf9f39b6a"}
Jul 09 18:05:44.924146 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.924048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6846c96d87-7mhdw" event={"ID":"f6363dd9-59ec-4cbd-ae82-a9e896a0814c","Type":"ContainerStarted","Data":"4774dbeb8ef94a9471242f8379c6b1141d5f66c682c82c013d7d0f6f8098295c"}
Jul 09 18:05:44.924526 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.924486    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:05:44.942556 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.942492    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-675d564c74-s5fsx" podStartSLOduration=50.942468002 podStartE2EDuration="50.942468002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:44.940675928 +0000 UTC m=+68.127653709" watchObservedRunningTime="2026-07-09 18:05:44.942468002 +0000 UTC m=+68.129445772"
Jul 09 18:05:44.961266 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.961192    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-kln87" podStartSLOduration=37.239208354 podStartE2EDuration="50.961164613s" totalImagesPullingTime="13.721956259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:44.961149581 +0000 UTC m=+68.148127349" watchObservedRunningTime="2026-07-09 18:05:44.961164613 +0000 UTC m=+68.148142383"
Jul 09 18:05:44.987488 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:44.987436    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6846c96d87-7mhdw" podStartSLOduration=66.987418375 podStartE2EDuration="1m6.987418375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:44.986698193 +0000 UTC m=+68.173675962" watchObservedRunningTime="2026-07-09 18:05:44.987418375 +0000 UTC m=+68.174396143"
Jul 09 18:05:45.035829 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:45.035772    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:45.159539 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:45.159420    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-kln87"
Jul 09 18:05:45.192075 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:45.191973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:45.383532 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:45.383498    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:45.479597 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:45.479502    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:45.492368 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:45.491233    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:45.492368 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:45.491300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:45.492368 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:45.491923    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:45.492368 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:05:45.491999    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls podName:f7b733c6-d68b-4be5-af36-4c7f7d8cddf0 nodeName:}" failed. No retries permitted until 2026-07-09 18:06:01.491977239 +0000 UTC m=+84.678954988 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnhg8" (UID: "f7b733c6-d68b-4be5-af36-4c7f7d8cddf0") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:45.492368 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:45.492024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:05:45.654077 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:45.654036    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:45.856615 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:45.856513    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:45.876692 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:45.876651    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:45.879844 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:45.879570    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:45.929007 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:45.928824    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:45.930700 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:45.930673    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-675d564c74-s5fsx"
Jul 09 18:05:46.055279 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:46.055242    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.138854 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:46.138729    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.276761 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:46.276728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.364744 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:46.364672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.445148 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:46.445067    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.525050 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:46.525011    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.924479 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:46.924446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.934550 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:46.934329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng" event={"ID":"4b9b442e-e7e5-4b54-bd16-5141660c77ce","Type":"ContainerStarted","Data":"c454b05eae0cc8aa0407b2cc192a181b1c67f59cc420351b5d27e6d9eda94062"}
Jul 09 18:05:46.934550 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:46.934393    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng" event={"ID":"4b9b442e-e7e5-4b54-bd16-5141660c77ce","Type":"ContainerStarted","Data":"4308782ad165ac01593c4f8253a86b711f8672582d200f63d13650c3504f934d"}
Jul 09 18:05:46.936422 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:46.936390    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p7zkv" event={"ID":"122683df-b7fe-4c5a-9974-987713ba1ee2","Type":"ContainerStarted","Data":"0a5f0920ac3f833c334055106e61175d9687ab1a513b174d79df6e9f81654b06"}
Jul 09 18:05:46.936573 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:46.936446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p7zkv" event={"ID":"122683df-b7fe-4c5a-9974-987713ba1ee2","Type":"ContainerStarted","Data":"a6bf0ce6b15e5760ee633d32377996b862167e1d0536a9ff4baab4904bd51b20"}
Jul 09 18:05:46.936705 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:46.936681    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:05:46.956181 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:46.956073    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2wnng" podStartSLOduration=50.5792228 podStartE2EDuration="52.956055671s" totalImagesPullingTime="2.376832871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:46.955029334 +0000 UTC m=+70.142007128" watchObservedRunningTime="2026-07-09 18:05:46.956055671 +0000 UTC m=+70.143033429"
Jul 09 18:05:46.974480 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:46.974415    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-p7zkv" podStartSLOduration=33.521121317 podStartE2EDuration="35.974397779s" totalImagesPullingTime="2.453276462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:46.974030989 +0000 UTC m=+70.161008811" watchObservedRunningTime="2026-07-09 18:05:46.974397779 +0000 UTC m=+70.161375547"
Jul 09 18:05:47.059254 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:47.059214    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:47.161555 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:47.161521    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:47.270304 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:47.270211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:47.444724 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:47.444684    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:47.838824 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:47.838758    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:48.247613 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:48.247565    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:48.435717 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:48.435683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:48.694639 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:48.694603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:48.771573 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:48.771538    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:48.946177 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:48.946144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5kvp7" event={"ID":"d3da572d-acbd-4c3c-a693-c0eac330f719","Type":"ContainerStarted","Data":"cac800da1e15bd7b89796abbf24052cf723341c36a54bcca17bb012629ce3085"}
Jul 09 18:05:48.969477 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:48.969422    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5kvp7" podStartSLOduration=66.881050659 podStartE2EDuration="1m11.969403355s" totalImagesPullingTime="5.088352696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:48.968660135 +0000 UTC m=+72.155637904" watchObservedRunningTime="2026-07-09 18:05:48.969403355 +0000 UTC m=+72.156381179"
Jul 09 18:05:52.163349 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:52.163311    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:54.892277 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:54.892234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:54.993327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:54.993291    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.183293 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:56.183261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.301668 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:56.301587    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.943535 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:05:56.943502    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p7zkv"
Jul 09 18:06:00.670450 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:00.670280    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:00.751239 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:00.751172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:01.539124 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:01.539086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:06:01.541444 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:01.541416    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b733c6-d68b-4be5-af36-4c7f7d8cddf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnhg8\" (UID: \"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0\") " pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:06:01.732157 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:01.732121    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qh478\""
Jul 09 18:06:01.739808 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:01.739767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dnhg8"
Jul 09 18:06:01.915820 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:01.915776    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dnhg8"]
Jul 09 18:06:01.918366 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:06:01.918335    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b733c6_d68b_4be5_af36_4c7f7d8cddf0.slice/crio-28ec94605cdd34c59c40fcc14330440e48d8ec402718caf99b9243fcae034931 WatchSource:0}: Error finding container 28ec94605cdd34c59c40fcc14330440e48d8ec402718caf99b9243fcae034931: Status 404 returned error can't find the container with id 28ec94605cdd34c59c40fcc14330440e48d8ec402718caf99b9243fcae034931
Jul 09 18:06:01.991758 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:01.991719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnhg8" event={"ID":"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0","Type":"ContainerStarted","Data":"28ec94605cdd34c59c40fcc14330440e48d8ec402718caf99b9243fcae034931"}
Jul 09 18:06:03.902991 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:03.902953    2578 patch_prober.go:28] interesting pod/image-registry-6846c96d87-7mhdw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 18:06:03.903416 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:03.903013    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6846c96d87-7mhdw" podUID="f6363dd9-59ec-4cbd-ae82-a9e896a0814c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:06:04.467433 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:04.467400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:04.559092 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:04.558995    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:05.933896 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:05.933854    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:06:06.010643 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:06.010587    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnhg8" event={"ID":"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0","Type":"ContainerStarted","Data":"3eb66a5d7115246107932f53d7fe8510642de81ca1957b8bb4cee1d624ee25f5"}
Jul 09 18:06:06.193768 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:06.193687    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:06.294869 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:06.294833    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:06.972447 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:06.972415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:07.017876 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:07.017834    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnhg8" event={"ID":"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0","Type":"ContainerStarted","Data":"19e260bc69b13011b9110bd3eb8893af42aba4b4f936589a4b3278ad4e07a0bd"}
Jul 09 18:06:07.092239 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:07.092192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:07.233234 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:07.233110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:07.675538 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:07.675455    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:07.769350 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:07.769314    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:10.024560 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:10.024526    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:10.036512 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:10.036469    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnhg8" event={"ID":"f7b733c6-d68b-4be5-af36-4c7f7d8cddf0","Type":"ContainerStarted","Data":"ee0530c633274f1b8e1283481d9fbee73c245ab37a8dcac63236e8029f7a67e1"}
Jul 09 18:06:10.059528 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:10.059473    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dnhg8" podStartSLOduration=34.123367286 podStartE2EDuration="41.059455354s" totalImagesPullingTime="6.936088068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:10.057683012 +0000 UTC m=+93.244660793" watchObservedRunningTime="2026-07-09 18:06:10.059455354 +0000 UTC m=+93.246433143"
Jul 09 18:06:10.125498 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:10.125461    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:10.557442 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:10.557397    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:10.657355 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:10.657323    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:14.927364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:14.927336    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f6494"
Jul 09 18:06:15.143018 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:15.142979    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:15.242101 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:15.242067    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:15.862979 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:15.862940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:06:15.865465 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:15.865442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d25c758-d9bb-4113-a71d-de846c62e5ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-cjt5m\" (UID: \"1d25c758-d9bb-4113-a71d-de846c62e5ef\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:06:15.963748 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:15.963705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9zf8g\" (UID: \"fccda6c1-c857-41be-a4fd-198cbd82a2c5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:06:15.966316 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:15.966287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fccda6c1-c857-41be-a4fd-198cbd82a2c5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9zf8g\" (UID: \"fccda6c1-c857-41be-a4fd-198cbd82a2c5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:06:15.979997 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:15.979967    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nt6vh\""
Jul 09 18:06:15.987563 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:15.987538    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"
Jul 09 18:06:16.074621 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:16.074596    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7hp5r\""
Jul 09 18:06:16.082421 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:16.082382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"
Jul 09 18:06:16.135037 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:16.134957    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m"]
Jul 09 18:06:16.140111 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:06:16.140073    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d25c758_d9bb_4113_a71d_de846c62e5ef.slice/crio-ef81f0797da55a5cb8bbb92884a540815bc096d5114d12e364c73b8e45da4f94 WatchSource:0}: Error finding container ef81f0797da55a5cb8bbb92884a540815bc096d5114d12e364c73b8e45da4f94: Status 404 returned error can't find the container with id ef81f0797da55a5cb8bbb92884a540815bc096d5114d12e364c73b8e45da4f94
Jul 09 18:06:16.227410 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:16.227372    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-9zf8g"]
Jul 09 18:06:16.230324 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:06:16.230294    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccda6c1_c857_41be_a4fd_198cbd82a2c5.slice/crio-503f939f16cc2c7c6951e2b4dbce4dcec22c839b9c24f0805616253247ea8701 WatchSource:0}: Error finding container 503f939f16cc2c7c6951e2b4dbce4dcec22c839b9c24f0805616253247ea8701: Status 404 returned error can't find the container with id 503f939f16cc2c7c6951e2b4dbce4dcec22c839b9c24f0805616253247ea8701
Jul 09 18:06:16.729976 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:16.729940    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:16.831604 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:16.831545    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:17.064610 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:17.064518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g" event={"ID":"fccda6c1-c857-41be-a4fd-198cbd82a2c5","Type":"ContainerStarted","Data":"503f939f16cc2c7c6951e2b4dbce4dcec22c839b9c24f0805616253247ea8701"}
Jul 09 18:06:17.065715 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:17.065689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m" event={"ID":"1d25c758-d9bb-4113-a71d-de846c62e5ef","Type":"ContainerStarted","Data":"ef81f0797da55a5cb8bbb92884a540815bc096d5114d12e364c73b8e45da4f94"}
Jul 09 18:06:17.274175 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:17.274129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:17.394988 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:17.394900    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:18.070659 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:18.070623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g" event={"ID":"fccda6c1-c857-41be-a4fd-198cbd82a2c5","Type":"ContainerStarted","Data":"ac0b69054dfaccc841e120dca01f371b60fdd2f91751fbcb0b2ba4d0008d2812"}
Jul 09 18:06:18.089720 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:18.089665    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-9zf8g" podStartSLOduration=73.782653528 podStartE2EDuration="1m15.089647885s" totalImagesPullingTime="1.306994357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:18.088900565 +0000 UTC m=+101.275878334" watchObservedRunningTime="2026-07-09 18:06:18.089647885 +0000 UTC m=+101.276625653"
Jul 09 18:06:18.295743 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:18.295701    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6846c96d87-7mhdw"]
Jul 09 18:06:20.061045 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:20.060955    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:20.080254 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:20.080204    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m" event={"ID":"1d25c758-d9bb-4113-a71d-de846c62e5ef","Type":"ContainerStarted","Data":"2048ae489f1628d62fe5959f4121158cedaa540715664b94158fdda8005ebfc5"}
Jul 09 18:06:20.102647 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:20.102592    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-cjt5m" podStartSLOduration=82.530996438 podStartE2EDuration="1m26.10257448s" totalImagesPullingTime="3.571578042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:20.101309581 +0000 UTC m=+103.288287368" watchObservedRunningTime="2026-07-09 18:06:20.10257448 +0000 UTC m=+103.289552248"
Jul 09 18:06:20.184537 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:20.184504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:20.278429 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:20.278390    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:20.384566 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:20.384481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:21.539598 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:21.539561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:21.627278 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:21.627243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:24.371816 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:24.371748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:24.565844 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:24.565813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:24.698481 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:24.698448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:24.803170 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:24.803142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:24.927381 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:24.927351    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:25.030769 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:25.030689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:26.274288 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:26.274258    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:28.567468 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.567414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:28.690188 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.690149    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:28.875725 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.875611    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g7js5"]
Jul 09 18:06:28.878067 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.878049    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:28.881193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.881169    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 18:06:28.881393 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.881349    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 18:06:28.881499 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.881482    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hpgbg\""
Jul 09 18:06:28.882378 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.882361    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 18:06:28.882481 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.882363    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 18:06:28.890071 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.890043    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:28.978018 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.977986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b635a0d-0095-471e-8ff4-d47c7a234223-sys\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:28.978214 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.978032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b635a0d-0095-471e-8ff4-d47c7a234223-root\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:28.978214 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.978061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-wtmp\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:28.978214 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.978088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-textfile\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:28.978214 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.978163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-tls\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:28.978214 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.978210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrxwm\" (UniqueName: \"kubernetes.io/projected/3b635a0d-0095-471e-8ff4-d47c7a234223-kube-api-access-rrxwm\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:28.978417 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.978246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:28.978417 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.978321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-accelerators-collector-config\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:28.978417 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:28.978352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b635a0d-0095-471e-8ff4-d47c7a234223-metrics-client-ca\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.016697 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.016664    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:29.079147 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.079106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b635a0d-0095-471e-8ff4-d47c7a234223-root\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.079147 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.079149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-wtmp\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.079393 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.079171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-textfile\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.079393 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.079238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b635a0d-0095-471e-8ff4-d47c7a234223-root\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.079393 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.079299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-tls\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.079393 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.079346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrxwm\" (UniqueName: \"kubernetes.io/projected/3b635a0d-0095-471e-8ff4-d47c7a234223-kube-api-access-rrxwm\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.079393 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.079302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-wtmp\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.079393 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.079371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.079659 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.079459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-accelerators-collector-config\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.079659 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.079495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b635a0d-0095-471e-8ff4-d47c7a234223-metrics-client-ca\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.079659 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.079509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-textfile\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.079659 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.079553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b635a0d-0095-471e-8ff4-d47c7a234223-sys\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.079659 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.079631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b635a0d-0095-471e-8ff4-d47c7a234223-sys\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.080076 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.080046    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-accelerators-collector-config\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.080192 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.080172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b635a0d-0095-471e-8ff4-d47c7a234223-metrics-client-ca\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.081648 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.081630    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-tls\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.081736 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.081682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b635a0d-0095-471e-8ff4-d47c7a234223-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.088662 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.088637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrxwm\" (UniqueName: \"kubernetes.io/projected/3b635a0d-0095-471e-8ff4-d47c7a234223-kube-api-access-rrxwm\") pod \"node-exporter-g7js5\" (UID: \"3b635a0d-0095-471e-8ff4-d47c7a234223\") " pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.129028 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.128935    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:29.187482 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.187448    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g7js5"
Jul 09 18:06:29.197847 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:06:29.197807    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b635a0d_0095_471e_8ff4_d47c7a234223.slice/crio-aca566f8c22a60b1e8e01c85986637a71b236f0c49521f28e702a95762f27ed6 WatchSource:0}: Error finding container aca566f8c22a60b1e8e01c85986637a71b236f0c49521f28e702a95762f27ed6: Status 404 returned error can't find the container with id aca566f8c22a60b1e8e01c85986637a71b236f0c49521f28e702a95762f27ed6
Jul 09 18:06:29.268248 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.268212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:29.589090 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.589054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:29.877098 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.877008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:29.987349 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:29.987312    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:30.115989 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:30.115955    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g7js5" event={"ID":"3b635a0d-0095-471e-8ff4-d47c7a234223","Type":"ContainerStarted","Data":"aca566f8c22a60b1e8e01c85986637a71b236f0c49521f28e702a95762f27ed6"}
Jul 09 18:06:30.188209 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:30.188128    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:31.121704 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:31.121667    2578 generic.go:356] "Generic (PLEG): container finished" podID="3b635a0d-0095-471e-8ff4-d47c7a234223" containerID="f1abd59b667364792a5b309f90173971ece24b06f86ae602569df790d36609ec" exitCode=0
Jul 09 18:06:31.122131 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:31.121738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g7js5" event={"ID":"3b635a0d-0095-471e-8ff4-d47c7a234223","Type":"ContainerDied","Data":"f1abd59b667364792a5b309f90173971ece24b06f86ae602569df790d36609ec"}
Jul 09 18:06:31.853011 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:31.852977    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:31.948158 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:31.948123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:32.046140 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:32.046107    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:32.128490 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:32.128392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g7js5" event={"ID":"3b635a0d-0095-471e-8ff4-d47c7a234223","Type":"ContainerStarted","Data":"5ded511255b49a59382175455d68b414ec8653f4edd6217fee5eefadd31dab27"}
Jul 09 18:06:32.128490 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:32.128456    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g7js5" event={"ID":"3b635a0d-0095-471e-8ff4-d47c7a234223","Type":"ContainerStarted","Data":"d953ac32e583e081bf6de2d14ce38029133aa3a03d959098ceb96b431875e2a0"}
Jul 09 18:06:32.157424 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:32.157370    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g7js5" podStartSLOduration=2.808999245 podStartE2EDuration="4.157352029s" totalImagesPullingTime="1.278873893s" totalInitContainerRuntime="69.478891ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:32.15628584 +0000 UTC m=+115.343263616" watchObservedRunningTime="2026-07-09 18:06:32.157352029 +0000 UTC m=+115.344329797"
Jul 09 18:06:32.210171 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:32.210139    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:33.560462 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:33.560434    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:33.656308 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:33.656252    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:33.767554 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:33.767516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:35.118250 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:35.118211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:39.894743 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:39.894710    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:39.997116 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:39.997082    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:41.405645 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:41.405611    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:41.474567 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:41.474539    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:42.215498 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:42.215458    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6" podUID="12137a00-1ea6-420d-84be-97efa63bffe8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 18:06:43.326934 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.326885    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6846c96d87-7mhdw" podUID="f6363dd9-59ec-4cbd-ae82-a9e896a0814c" containerName="registry" containerID="cri-o://0bdc4b370f9b30866d6cbd2511d6ddad96a02f8928e4b7694eaea58cf9f39b6a" gracePeriod=30
Jul 09 18:06:43.568661 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.568634    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:06:43.717193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.717165    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-bound-sa-token\") pod \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") "
Jul 09 18:06:43.717361 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.717205    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls\") pod \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") "
Jul 09 18:06:43.717361 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.717252    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-certificates\") pod \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") "
Jul 09 18:06:43.717361 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.717274    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-ca-trust-extracted\") pod \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") "
Jul 09 18:06:43.717361 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.717298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-trusted-ca\") pod \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") "
Jul 09 18:06:43.717361 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.717337    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gkx7\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-kube-api-access-6gkx7\") pod \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") "
Jul 09 18:06:43.717602 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.717376    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-installation-pull-secrets\") pod \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") "
Jul 09 18:06:43.717602 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.717424    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-image-registry-private-configuration\") pod \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\" (UID: \"f6363dd9-59ec-4cbd-ae82-a9e896a0814c\") "
Jul 09 18:06:43.717829 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.717762    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f6363dd9-59ec-4cbd-ae82-a9e896a0814c" (UID: "f6363dd9-59ec-4cbd-ae82-a9e896a0814c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:43.718280 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.718235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f6363dd9-59ec-4cbd-ae82-a9e896a0814c" (UID: "f6363dd9-59ec-4cbd-ae82-a9e896a0814c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:43.720271 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.720232    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f6363dd9-59ec-4cbd-ae82-a9e896a0814c" (UID: "f6363dd9-59ec-4cbd-ae82-a9e896a0814c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:43.720386 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.720326    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-kube-api-access-6gkx7" (OuterVolumeSpecName: "kube-api-access-6gkx7") pod "f6363dd9-59ec-4cbd-ae82-a9e896a0814c" (UID: "f6363dd9-59ec-4cbd-ae82-a9e896a0814c"). InnerVolumeSpecName "kube-api-access-6gkx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:43.720451 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.720433    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f6363dd9-59ec-4cbd-ae82-a9e896a0814c" (UID: "f6363dd9-59ec-4cbd-ae82-a9e896a0814c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:43.720551 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.720473    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f6363dd9-59ec-4cbd-ae82-a9e896a0814c" (UID: "f6363dd9-59ec-4cbd-ae82-a9e896a0814c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:43.720604 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.720548    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f6363dd9-59ec-4cbd-ae82-a9e896a0814c" (UID: "f6363dd9-59ec-4cbd-ae82-a9e896a0814c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:43.726370 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.726342    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f6363dd9-59ec-4cbd-ae82-a9e896a0814c" (UID: "f6363dd9-59ec-4cbd-ae82-a9e896a0814c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:06:43.818932 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.818895    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-installation-pull-secrets\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.818932 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.818925    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-image-registry-private-configuration\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.818932 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.818935    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-bound-sa-token\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.818932 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.818944    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-tls\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.819197 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.818953    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-registry-certificates\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.819197 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.818962    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-ca-trust-extracted\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.819197 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.818970    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-trusted-ca\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.819197 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:43.818978    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gkx7\" (UniqueName: \"kubernetes.io/projected/f6363dd9-59ec-4cbd-ae82-a9e896a0814c-kube-api-access-6gkx7\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:06:44.175726 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:44.175628    2578 generic.go:356] "Generic (PLEG): container finished" podID="f6363dd9-59ec-4cbd-ae82-a9e896a0814c" containerID="0bdc4b370f9b30866d6cbd2511d6ddad96a02f8928e4b7694eaea58cf9f39b6a" exitCode=0
Jul 09 18:06:44.175726 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:44.175679    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6846c96d87-7mhdw"
Jul 09 18:06:44.175726 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:44.175712    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6846c96d87-7mhdw" event={"ID":"f6363dd9-59ec-4cbd-ae82-a9e896a0814c","Type":"ContainerDied","Data":"0bdc4b370f9b30866d6cbd2511d6ddad96a02f8928e4b7694eaea58cf9f39b6a"}
Jul 09 18:06:44.175989 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:44.175769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6846c96d87-7mhdw" event={"ID":"f6363dd9-59ec-4cbd-ae82-a9e896a0814c","Type":"ContainerDied","Data":"4774dbeb8ef94a9471242f8379c6b1141d5f66c682c82c013d7d0f6f8098295c"}
Jul 09 18:06:44.175989 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:44.175788    2578 scope.go:117] "RemoveContainer" containerID="0bdc4b370f9b30866d6cbd2511d6ddad96a02f8928e4b7694eaea58cf9f39b6a"
Jul 09 18:06:44.184896 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:44.184875    2578 scope.go:117] "RemoveContainer" containerID="0bdc4b370f9b30866d6cbd2511d6ddad96a02f8928e4b7694eaea58cf9f39b6a"
Jul 09 18:06:44.185202 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:06:44.185183    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bdc4b370f9b30866d6cbd2511d6ddad96a02f8928e4b7694eaea58cf9f39b6a\": container with ID starting with 0bdc4b370f9b30866d6cbd2511d6ddad96a02f8928e4b7694eaea58cf9f39b6a not found: ID does not exist" containerID="0bdc4b370f9b30866d6cbd2511d6ddad96a02f8928e4b7694eaea58cf9f39b6a"
Jul 09 18:06:44.185267 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:44.185215    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bdc4b370f9b30866d6cbd2511d6ddad96a02f8928e4b7694eaea58cf9f39b6a"} err="failed to get container status \"0bdc4b370f9b30866d6cbd2511d6ddad96a02f8928e4b7694eaea58cf9f39b6a\": rpc error: code = NotFound desc = could not find container \"0bdc4b370f9b30866d6cbd2511d6ddad96a02f8928e4b7694eaea58cf9f39b6a\": container with ID starting with 0bdc4b370f9b30866d6cbd2511d6ddad96a02f8928e4b7694eaea58cf9f39b6a not found: ID does not exist"
Jul 09 18:06:44.202485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:44.202448    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6846c96d87-7mhdw"]
Jul 09 18:06:44.205448 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:44.205418    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6846c96d87-7mhdw"]
Jul 09 18:06:45.402936 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:45.402897    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6363dd9-59ec-4cbd-ae82-a9e896a0814c" path="/var/lib/kubelet/pods/f6363dd9-59ec-4cbd-ae82-a9e896a0814c/volumes"
Jul 09 18:06:47.249975 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:47.249933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs\") pod \"network-metrics-daemon-kpcpn\" (UID: \"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58\") " pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:06:47.252335 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:47.252313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58-metrics-certs\") pod \"network-metrics-daemon-kpcpn\" (UID: \"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58\") " pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:06:47.527427 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:47.527339    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhwtc\""
Jul 09 18:06:47.535580 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:47.535538    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpcpn"
Jul 09 18:06:47.675933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:47.675896    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kpcpn"]
Jul 09 18:06:47.679876 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:06:47.679843    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a6a28b_c7c4_4cb3_b25b_3d777e5aeb58.slice/crio-897e92942ca97f3697fb1ffc7d17c9dfe524e658e1cd1c1a1b03e28ab22216ed WatchSource:0}: Error finding container 897e92942ca97f3697fb1ffc7d17c9dfe524e658e1cd1c1a1b03e28ab22216ed: Status 404 returned error can't find the container with id 897e92942ca97f3697fb1ffc7d17c9dfe524e658e1cd1c1a1b03e28ab22216ed
Jul 09 18:06:48.193281 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:48.193243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kpcpn" event={"ID":"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58","Type":"ContainerStarted","Data":"897e92942ca97f3697fb1ffc7d17c9dfe524e658e1cd1c1a1b03e28ab22216ed"}
Jul 09 18:06:49.950000 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:49.949963    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:50.053133 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:50.053099    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:50.118638 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:50.118609    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:51.210594 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:51.210554    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kpcpn" event={"ID":"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58","Type":"ContainerStarted","Data":"25112f9340f71834d4af24e0a96d24251fe68853bea4d9ead20f6aedf018f6fa"}
Jul 09 18:06:51.211032 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:51.210640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kpcpn" event={"ID":"11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58","Type":"ContainerStarted","Data":"60df6a474467d502047460ac868124823fae5fb01c4e9f9b509d6943d5c69649"}
Jul 09 18:06:51.228283 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:51.228230    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kpcpn" podStartSLOduration=130.918275371 podStartE2EDuration="2m14.228213515s" totalImagesPullingTime="3.309938144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:51.227844873 +0000 UTC m=+134.414822661" watchObservedRunningTime="2026-07-09 18:06:51.228213515 +0000 UTC m=+134.415191282"
Jul 09 18:06:51.345124 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:51.345037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:51.422385 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:51.422351    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:52.215356 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:52.215321    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6" podUID="12137a00-1ea6-420d-84be-97efa63bffe8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 18:06:54.225466 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:54.225432    2578 generic.go:356] "Generic (PLEG): container finished" podID="5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b" containerID="cc37c16997233e28d4185f05ad8e988fa03358ea086a2434a87ecd05eb2bd099" exitCode=0
Jul 09 18:06:54.225947 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:54.225506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k" event={"ID":"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b","Type":"ContainerDied","Data":"cc37c16997233e28d4185f05ad8e988fa03358ea086a2434a87ecd05eb2bd099"}
Jul 09 18:06:54.225947 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:54.225917    2578 scope.go:117] "RemoveContainer" containerID="cc37c16997233e28d4185f05ad8e988fa03358ea086a2434a87ecd05eb2bd099"
Jul 09 18:06:54.227639 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:54.227595    2578 generic.go:356] "Generic (PLEG): container finished" podID="e815768d-07d5-486e-81c3-d29a321181a0" containerID="ec2ad9a37110a512758e4bfc6fdddf497c5af9de706f7f391251575e94e0e745" exitCode=0
Jul 09 18:06:54.227695 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:54.227663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw" event={"ID":"e815768d-07d5-486e-81c3-d29a321181a0","Type":"ContainerDied","Data":"ec2ad9a37110a512758e4bfc6fdddf497c5af9de706f7f391251575e94e0e745"}
Jul 09 18:06:54.227984 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:54.227969    2578 scope.go:117] "RemoveContainer" containerID="ec2ad9a37110a512758e4bfc6fdddf497c5af9de706f7f391251575e94e0e745"
Jul 09 18:06:55.233827 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:55.233765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-ccd5k" event={"ID":"5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b","Type":"ContainerStarted","Data":"08402806a2d2143072e4e3a5092e09a736eb34415d5a28c184e9ace8e0350037"}
Jul 09 18:06:55.235692 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:55.235659    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-87nrw" event={"ID":"e815768d-07d5-486e-81c3-d29a321181a0","Type":"ContainerStarted","Data":"230eb4f8f878c97affd95dab00cc001632f0bc127bfd7f8e2fada44e8493897f"}
Jul 09 18:06:59.251924 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:59.251891    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e" containerID="788c857370fea05ae4aee3e378036dcf18f8e819bba801caa7ca0a1ce0ed39c3" exitCode=0
Jul 09 18:06:59.252403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:59.251968    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz" event={"ID":"1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e","Type":"ContainerDied","Data":"788c857370fea05ae4aee3e378036dcf18f8e819bba801caa7ca0a1ce0ed39c3"}
Jul 09 18:06:59.252403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:06:59.252369    2578 scope.go:117] "RemoveContainer" containerID="788c857370fea05ae4aee3e378036dcf18f8e819bba801caa7ca0a1ce0ed39c3"
Jul 09 18:07:00.262004 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:00.261967    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-54gvz" event={"ID":"1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e","Type":"ContainerStarted","Data":"2524c6cd5dd674a875e1e1c5af169d8da8db5747efc757f57d643d78d501b51a"}
Jul 09 18:07:02.215391 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:02.215350    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6" podUID="12137a00-1ea6-420d-84be-97efa63bffe8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 18:07:02.215880 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:02.215436    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6"
Jul 09 18:07:02.215959 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:02.215941    2578 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"70f80621f49c4c66e7109bb30f9407029e76784cf0b20a0801ae5cd27d934fa9"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 09 18:07:02.215999 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:02.215981    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6" podUID="12137a00-1ea6-420d-84be-97efa63bffe8" containerName="service-proxy" containerID="cri-o://70f80621f49c4c66e7109bb30f9407029e76784cf0b20a0801ae5cd27d934fa9" gracePeriod=30
Jul 09 18:07:03.276030 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:03.275995    2578 generic.go:356] "Generic (PLEG): container finished" podID="12137a00-1ea6-420d-84be-97efa63bffe8" containerID="70f80621f49c4c66e7109bb30f9407029e76784cf0b20a0801ae5cd27d934fa9" exitCode=2
Jul 09 18:07:03.276412 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:03.276043    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6" event={"ID":"12137a00-1ea6-420d-84be-97efa63bffe8","Type":"ContainerDied","Data":"70f80621f49c4c66e7109bb30f9407029e76784cf0b20a0801ae5cd27d934fa9"}
Jul 09 18:07:03.276412 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:03.276076    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74f75ddc54-cncf6" event={"ID":"12137a00-1ea6-420d-84be-97efa63bffe8","Type":"ContainerStarted","Data":"6d2aff9f2296bd73afc776ddde7b9e5dce666f3a7d01499946a6b319bfbfe5f2"}
Jul 09 18:07:45.038879 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:45.038826    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:45.139691 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:45.139657    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:45.449903 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:45.449874    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:45.508778 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:45.508741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:56.209293 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:56.209254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:56.309166 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:56.309137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:56.378222 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:56.378191    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:57.021395 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:57.021362    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:57.087503 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:07:57.087454    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:09.909758 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:08:09.909717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:10.011948 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:08:10.011914    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:29.716054 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:08:29.716016    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:36.177587 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:08:36.177552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:36.265715 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:08:36.265676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:36.529953 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:08:36.529915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:36.624368 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:08:36.624333    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:36.726286 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:08:36.726253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:59.066286 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:08:59.066190    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:59.183996 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:08:59.183962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:05.136911 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:05.136875    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:05.449266 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:05.449236    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:05.661966 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:05.661930    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:05.765283 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:05.765195    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:06.014831 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:06.014786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:06.180375 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:06.180294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:06.288765 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:06.288733    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:06.384761 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:06.384729    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:06.593156 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:06.593100    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:06.687857 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:06.687822    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:06.896253 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:06.896168    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:07.090962 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:07.090927    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:07.196485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:07.196446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:37.213920 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:37.213884    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:09:37.214496 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:37.213972    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:09:37.223292 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:37.223269    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:09:37.223464 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:37.223367    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:09:37.226269 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:09:37.226251    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 18:10:19.611341 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:10:19.611305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:21.212989 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:10:21.212947    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:52.765397 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:10:52.765299    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:52.864688 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:10:52.864644    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:57.918783 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:10:57.918739    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:58.024971 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:10:58.024939    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:05.567951 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:05.567911    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:05.672773 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:05.672737    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:09.678969 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:09.678929    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:09.776753 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:09.776718    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:10.935896 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:10.935860    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:11.141701 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:11.141669    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:11.892868 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:11.892828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:13.056786 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:13.056749    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:13.602303 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:13.602264    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:14.473724 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:14.473689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:14.610364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:14.610320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:14.744026 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:14.743941    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:21.120865 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:21.120823    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:27.597068 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:27.597031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:27.701361 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:27.701326    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:28.945633 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:28.945575    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6d8k"]
Jul 09 18:11:28.946035 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:28.945898    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6363dd9-59ec-4cbd-ae82-a9e896a0814c" containerName="registry"
Jul 09 18:11:28.946035 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:28.945909    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6363dd9-59ec-4cbd-ae82-a9e896a0814c" containerName="registry"
Jul 09 18:11:28.946035 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:28.945972    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6363dd9-59ec-4cbd-ae82-a9e896a0814c" containerName="registry"
Jul 09 18:11:28.948273 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:28.948248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6d8k"
Jul 09 18:11:28.951132 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:28.951108    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:11:28.951268 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:28.951153    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pf68x\""
Jul 09 18:11:28.952224 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:28.952199    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 09 18:11:28.959557 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:28.959523    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6d8k"]
Jul 09 18:11:29.010662 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:29.010617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6nt\" (UniqueName: \"kubernetes.io/projected/316c9b2c-218d-4cfe-a844-1a7f45025513-kube-api-access-ck6nt\") pod \"openshift-lws-operator-fd8ccff4c-x6d8k\" (UID: \"316c9b2c-218d-4cfe-a844-1a7f45025513\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6d8k"
Jul 09 18:11:29.010899 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:29.010736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/316c9b2c-218d-4cfe-a844-1a7f45025513-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x6d8k\" (UID: \"316c9b2c-218d-4cfe-a844-1a7f45025513\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6d8k"
Jul 09 18:11:29.111361 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:29.111322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6nt\" (UniqueName: \"kubernetes.io/projected/316c9b2c-218d-4cfe-a844-1a7f45025513-kube-api-access-ck6nt\") pod \"openshift-lws-operator-fd8ccff4c-x6d8k\" (UID: \"316c9b2c-218d-4cfe-a844-1a7f45025513\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6d8k"
Jul 09 18:11:29.111543 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:29.111383    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/316c9b2c-218d-4cfe-a844-1a7f45025513-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x6d8k\" (UID: \"316c9b2c-218d-4cfe-a844-1a7f45025513\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6d8k"
Jul 09 18:11:29.111777 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:29.111758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/316c9b2c-218d-4cfe-a844-1a7f45025513-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x6d8k\" (UID: \"316c9b2c-218d-4cfe-a844-1a7f45025513\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6d8k"
Jul 09 18:11:29.123352 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:29.123315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6nt\" (UniqueName: \"kubernetes.io/projected/316c9b2c-218d-4cfe-a844-1a7f45025513-kube-api-access-ck6nt\") pod \"openshift-lws-operator-fd8ccff4c-x6d8k\" (UID: \"316c9b2c-218d-4cfe-a844-1a7f45025513\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6d8k"
Jul 09 18:11:29.257950 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:29.257913    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6d8k"
Jul 09 18:11:29.408989 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:29.408954    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6d8k"]
Jul 09 18:11:29.413023 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:11:29.412993    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod316c9b2c_218d_4cfe_a844_1a7f45025513.slice/crio-b750f7c6b57ac034dace59ab8a28a9a51c12c7b9132e1a78060971f729208c13 WatchSource:0}: Error finding container b750f7c6b57ac034dace59ab8a28a9a51c12c7b9132e1a78060971f729208c13: Status 404 returned error can't find the container with id b750f7c6b57ac034dace59ab8a28a9a51c12c7b9132e1a78060971f729208c13
Jul 09 18:11:29.414529 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:29.414511    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:11:30.309206 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:30.309167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6d8k" event={"ID":"316c9b2c-218d-4cfe-a844-1a7f45025513","Type":"ContainerStarted","Data":"b750f7c6b57ac034dace59ab8a28a9a51c12c7b9132e1a78060971f729208c13"}
Jul 09 18:11:36.763578 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:36.763542    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:36.864441 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:36.864408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:44.015039 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:44.015006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:44.125946 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:44.125911    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:45.952282 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:45.952246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:46.048018 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:46.047978    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:56.413623 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:56.413584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6d8k" event={"ID":"316c9b2c-218d-4cfe-a844-1a7f45025513","Type":"ContainerStarted","Data":"f7aca87f53630eac9b20d9ba251b1211619636e8b9bf7d57c9860a4de96cbac4"}
Jul 09 18:11:56.433057 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:56.433001    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6d8k" podStartSLOduration=1.920437773 podStartE2EDuration="28.432983923s" totalImagesPullingTime="26.51254615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:56.432039205 +0000 UTC m=+439.619016995" watchObservedRunningTime="2026-07-09 18:11:56.432983923 +0000 UTC m=+439.619961691"
Jul 09 18:11:57.352740 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:57.352703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:57.444092 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:11:57.444054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:04.448542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:04.448449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:08.742001 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:08.741964    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:08.868303 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:08.868271    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"]
Jul 09 18:12:08.870666 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:08.870642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:08.874374 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:08.874346    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 09 18:12:08.874960 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:08.874931    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 09 18:12:08.875690 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:08.875665    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 09 18:12:08.875879 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:08.875858    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xndsb\""
Jul 09 18:12:08.881368 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:08.881310    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:08.892376 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:08.892344    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"]
Jul 09 18:12:08.940205 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:08.940157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d72e52ed-cf6e-486f-bb03-b336eb2272a0-manager-config\") pod \"lws-controller-manager-9fc7df7df-67cff\" (UID: \"d72e52ed-cf6e-486f-bb03-b336eb2272a0\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:08.940410 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:08.940225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d72e52ed-cf6e-486f-bb03-b336eb2272a0-metrics-cert\") pod \"lws-controller-manager-9fc7df7df-67cff\" (UID: \"d72e52ed-cf6e-486f-bb03-b336eb2272a0\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:08.940410 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:08.940280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xr79\" (UniqueName: \"kubernetes.io/projected/d72e52ed-cf6e-486f-bb03-b336eb2272a0-kube-api-access-9xr79\") pod \"lws-controller-manager-9fc7df7df-67cff\" (UID: \"d72e52ed-cf6e-486f-bb03-b336eb2272a0\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:08.940410 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:08.940313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d72e52ed-cf6e-486f-bb03-b336eb2272a0-cert\") pod \"lws-controller-manager-9fc7df7df-67cff\" (UID: \"d72e52ed-cf6e-486f-bb03-b336eb2272a0\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:09.041775 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:09.041682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xr79\" (UniqueName: \"kubernetes.io/projected/d72e52ed-cf6e-486f-bb03-b336eb2272a0-kube-api-access-9xr79\") pod \"lws-controller-manager-9fc7df7df-67cff\" (UID: \"d72e52ed-cf6e-486f-bb03-b336eb2272a0\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:09.041775 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:09.041725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d72e52ed-cf6e-486f-bb03-b336eb2272a0-cert\") pod \"lws-controller-manager-9fc7df7df-67cff\" (UID: \"d72e52ed-cf6e-486f-bb03-b336eb2272a0\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:09.041775 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:09.041759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d72e52ed-cf6e-486f-bb03-b336eb2272a0-manager-config\") pod \"lws-controller-manager-9fc7df7df-67cff\" (UID: \"d72e52ed-cf6e-486f-bb03-b336eb2272a0\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:09.042080 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:09.041828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d72e52ed-cf6e-486f-bb03-b336eb2272a0-metrics-cert\") pod \"lws-controller-manager-9fc7df7df-67cff\" (UID: \"d72e52ed-cf6e-486f-bb03-b336eb2272a0\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:09.042557 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:09.042536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d72e52ed-cf6e-486f-bb03-b336eb2272a0-manager-config\") pod \"lws-controller-manager-9fc7df7df-67cff\" (UID: \"d72e52ed-cf6e-486f-bb03-b336eb2272a0\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:09.044328 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:09.044302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d72e52ed-cf6e-486f-bb03-b336eb2272a0-metrics-cert\") pod \"lws-controller-manager-9fc7df7df-67cff\" (UID: \"d72e52ed-cf6e-486f-bb03-b336eb2272a0\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:09.044489 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:09.044465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d72e52ed-cf6e-486f-bb03-b336eb2272a0-cert\") pod \"lws-controller-manager-9fc7df7df-67cff\" (UID: \"d72e52ed-cf6e-486f-bb03-b336eb2272a0\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:09.052116 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:09.052084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xr79\" (UniqueName: \"kubernetes.io/projected/d72e52ed-cf6e-486f-bb03-b336eb2272a0-kube-api-access-9xr79\") pod \"lws-controller-manager-9fc7df7df-67cff\" (UID: \"d72e52ed-cf6e-486f-bb03-b336eb2272a0\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:09.186068 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:09.186025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:09.374225 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:09.374197    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"]
Jul 09 18:12:09.376858 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:12:09.376828    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72e52ed_cf6e_486f_bb03_b336eb2272a0.slice/crio-1aac1ff8b708634744eefa49b279ea17152e7780e62952dee83d252e574cabba WatchSource:0}: Error finding container 1aac1ff8b708634744eefa49b279ea17152e7780e62952dee83d252e574cabba: Status 404 returned error can't find the container with id 1aac1ff8b708634744eefa49b279ea17152e7780e62952dee83d252e574cabba
Jul 09 18:12:09.468274 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:09.468236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff" event={"ID":"d72e52ed-cf6e-486f-bb03-b336eb2272a0","Type":"ContainerStarted","Data":"1aac1ff8b708634744eefa49b279ea17152e7780e62952dee83d252e574cabba"}
Jul 09 18:12:10.935440 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:10.935344    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:11.074672 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:11.074633    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:11.477305 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:11.477271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff" event={"ID":"d72e52ed-cf6e-486f-bb03-b336eb2272a0","Type":"ContainerStarted","Data":"e68a649d7ee7dc27fdd1d773e453a9883e723861ac0fc32deb54dea13dc9b0d2"}
Jul 09 18:12:11.477494 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:11.477370    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:12.918951 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:12.918912    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:14.985192 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:14.985161    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:15.094580 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:15.094551    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:18.844369 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:18.844330    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:18.927192 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:18.927161    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:22.115718 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:22.115683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:22.483628 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:22.483600    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff"
Jul 09 18:12:22.504442 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:22.504390    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-67cff" podStartSLOduration=13.406311648 podStartE2EDuration="14.5043758s" totalImagesPullingTime="1.098064152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:12:11.509998731 +0000 UTC m=+454.696976505" watchObservedRunningTime="2026-07-09 18:12:22.5043758 +0000 UTC m=+465.691353568"
Jul 09 18:12:24.255576 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:24.255539    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:24.357993 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:24.357957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:37.361526 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:37.361492    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:38.140738 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:38.140705    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:41.280056 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:41.280023    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:41.429684 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:41.429645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:43.554223 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:43.554185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:43.674808 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:43.674743    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:45.716657 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:45.716625    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:45.832296 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:45.832264    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:46.389954 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:46.389915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:46.752922 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:46.752887    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:47.417497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:47.417462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:47.569839 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:47.569784    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:51.217065 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:51.217030    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:51.323562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:51.323524    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:51.788315 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:51.788275    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:51.892746 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:51.892705    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:54.180182 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:54.180143    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:54.280955 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:54.280919    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:56.777912 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:56.777877    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:56.873558 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:12:56.873524    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:02.009668 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.009627    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"]
Jul 09 18:13:02.011849 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.011828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"
Jul 09 18:13:02.032949 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.032911    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dxql7\""
Jul 09 18:13:02.033145 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.033061    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 09 18:13:02.033836 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.033817    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 09 18:13:02.033906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.033825    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 18:13:02.033906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.033822    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 18:13:02.060455 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.060423    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"]
Jul 09 18:13:02.062975 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.062940    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:02.070076 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.070041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a93127a-15c3-410c-9f2b-1102ee005f45-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-7mvzh\" (UID: \"0a93127a-15c3-410c-9f2b-1102ee005f45\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"
Jul 09 18:13:02.070283 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.070098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a93127a-15c3-410c-9f2b-1102ee005f45-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-7mvzh\" (UID: \"0a93127a-15c3-410c-9f2b-1102ee005f45\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"
Jul 09 18:13:02.070283 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.070182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2t2\" (UniqueName: \"kubernetes.io/projected/0a93127a-15c3-410c-9f2b-1102ee005f45-kube-api-access-2g2t2\") pod \"kuadrant-console-plugin-85fd4b8b9d-7mvzh\" (UID: \"0a93127a-15c3-410c-9f2b-1102ee005f45\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"
Jul 09 18:13:02.170866 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.170790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2t2\" (UniqueName: \"kubernetes.io/projected/0a93127a-15c3-410c-9f2b-1102ee005f45-kube-api-access-2g2t2\") pod \"kuadrant-console-plugin-85fd4b8b9d-7mvzh\" (UID: \"0a93127a-15c3-410c-9f2b-1102ee005f45\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"
Jul 09 18:13:02.171053 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.170905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a93127a-15c3-410c-9f2b-1102ee005f45-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-7mvzh\" (UID: \"0a93127a-15c3-410c-9f2b-1102ee005f45\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"
Jul 09 18:13:02.171053 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.170938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a93127a-15c3-410c-9f2b-1102ee005f45-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-7mvzh\" (UID: \"0a93127a-15c3-410c-9f2b-1102ee005f45\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"
Jul 09 18:13:02.171125 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:13:02.171050    2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 09 18:13:02.171125 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:13:02.171122    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a93127a-15c3-410c-9f2b-1102ee005f45-plugin-serving-cert podName:0a93127a-15c3-410c-9f2b-1102ee005f45 nodeName:}" failed. No retries permitted until 2026-07-09 18:13:02.671098836 +0000 UTC m=+505.858076584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/0a93127a-15c3-410c-9f2b-1102ee005f45-plugin-serving-cert") pod "kuadrant-console-plugin-85fd4b8b9d-7mvzh" (UID: "0a93127a-15c3-410c-9f2b-1102ee005f45") : secret "plugin-serving-cert" not found
Jul 09 18:13:02.171556 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.171536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a93127a-15c3-410c-9f2b-1102ee005f45-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-7mvzh\" (UID: \"0a93127a-15c3-410c-9f2b-1102ee005f45\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"
Jul 09 18:13:02.194760 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.194727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2t2\" (UniqueName: \"kubernetes.io/projected/0a93127a-15c3-410c-9f2b-1102ee005f45-kube-api-access-2g2t2\") pod \"kuadrant-console-plugin-85fd4b8b9d-7mvzh\" (UID: \"0a93127a-15c3-410c-9f2b-1102ee005f45\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"
Jul 09 18:13:02.466063 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.466031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:02.674986 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.674947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a93127a-15c3-410c-9f2b-1102ee005f45-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-7mvzh\" (UID: \"0a93127a-15c3-410c-9f2b-1102ee005f45\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"
Jul 09 18:13:02.677553 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.677527    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a93127a-15c3-410c-9f2b-1102ee005f45-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-7mvzh\" (UID: \"0a93127a-15c3-410c-9f2b-1102ee005f45\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"
Jul 09 18:13:02.921737 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:02.921637    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"
Jul 09 18:13:03.085397 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:03.085357    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh"]
Jul 09 18:13:03.089125 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:13:03.089098    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a93127a_15c3_410c_9f2b_1102ee005f45.slice/crio-157ff6c31287007bd75400cb8eeaaec69bfd132b885c04d8ef3bb2e795d6b67e WatchSource:0}: Error finding container 157ff6c31287007bd75400cb8eeaaec69bfd132b885c04d8ef3bb2e795d6b67e: Status 404 returned error can't find the container with id 157ff6c31287007bd75400cb8eeaaec69bfd132b885c04d8ef3bb2e795d6b67e
Jul 09 18:13:03.365443 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:03.365408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:03.556753 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:03.556718    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:03.685978 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:03.685887    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:03.688643 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:03.688608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh" event={"ID":"0a93127a-15c3-410c-9f2b-1102ee005f45","Type":"ContainerStarted","Data":"157ff6c31287007bd75400cb8eeaaec69bfd132b885c04d8ef3bb2e795d6b67e"}
Jul 09 18:13:08.715128 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:08.715089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh" event={"ID":"0a93127a-15c3-410c-9f2b-1102ee005f45","Type":"ContainerStarted","Data":"91cb710dc0d9d0b129f075bcef062154eaaaff3c0f132972c2ee37f3372f7149"}
Jul 09 18:13:08.735337 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:08.735272    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7mvzh" podStartSLOduration=3.01767777 podStartE2EDuration="7.735250454s" totalImagesPullingTime="4.717572684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:08.734392194 +0000 UTC m=+511.921369974" watchObservedRunningTime="2026-07-09 18:13:08.735250454 +0000 UTC m=+511.922228222"
Jul 09 18:13:10.589172 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:10.589139    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:10.768372 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:10.768336    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:10.873840 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:10.873740    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:14.172378 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:14.172339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:14.367514 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:14.367484    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:17.188570 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:17.188531    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:17.336198 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:17.336169    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:20.143022 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:20.142984    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:22.974967 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:22.974930    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:23.161820 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:23.161768    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:23.307987 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:23.307906    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:23.404080 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:23.404047    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:23.515036 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:23.515008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:42.234389 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:42.234309    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:42.405121 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:42.405089    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:42.533969 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:42.533877    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:43.190178 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.190140    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c89bb85-b8bml"]
Jul 09 18:13:43.194666 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.194639    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-b8bml"
Jul 09 18:13:43.199957 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.199933    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-swxmt\""
Jul 09 18:13:43.216528 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.216498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fls8\" (UniqueName: \"kubernetes.io/projected/25725ad9-41b6-4c06-af4e-48d4f37cbc60-kube-api-access-4fls8\") pod \"authorino-b6c89bb85-b8bml\" (UID: \"25725ad9-41b6-4c06-af4e-48d4f37cbc60\") " pod="kuadrant-system/authorino-b6c89bb85-b8bml"
Jul 09 18:13:43.240902 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.240865    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-b8bml"]
Jul 09 18:13:43.298606 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.298574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:43.317450 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.317409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fls8\" (UniqueName: \"kubernetes.io/projected/25725ad9-41b6-4c06-af4e-48d4f37cbc60-kube-api-access-4fls8\") pod \"authorino-b6c89bb85-b8bml\" (UID: \"25725ad9-41b6-4c06-af4e-48d4f37cbc60\") " pod="kuadrant-system/authorino-b6c89bb85-b8bml"
Jul 09 18:13:43.331008 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.330973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fls8\" (UniqueName: \"kubernetes.io/projected/25725ad9-41b6-4c06-af4e-48d4f37cbc60-kube-api-access-4fls8\") pod \"authorino-b6c89bb85-b8bml\" (UID: \"25725ad9-41b6-4c06-af4e-48d4f37cbc60\") " pod="kuadrant-system/authorino-b6c89bb85-b8bml"
Jul 09 18:13:43.457434 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.457402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:43.504352 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.504316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-b8bml"
Jul 09 18:13:43.597298 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.597234    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-lv8th"]
Jul 09 18:13:43.600723 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.600699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-lv8th"
Jul 09 18:13:43.620653 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.620614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmrmj\" (UniqueName: \"kubernetes.io/projected/3a1e2296-0a12-4284-a1ef-3e8b41a11556-kube-api-access-bmrmj\") pod \"authorino-8df697c9f-lv8th\" (UID: \"3a1e2296-0a12-4284-a1ef-3e8b41a11556\") " pod="kuadrant-system/authorino-8df697c9f-lv8th"
Jul 09 18:13:43.622200 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.622175    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-lv8th"]
Jul 09 18:13:43.665457 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.665424    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-b8bml"]
Jul 09 18:13:43.668503 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:13:43.668476    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25725ad9_41b6_4c06_af4e_48d4f37cbc60.slice/crio-74a00d7638b03a191b050a76f3e01ff248ea785ee4ce4435c9a784ea88398560 WatchSource:0}: Error finding container 74a00d7638b03a191b050a76f3e01ff248ea785ee4ce4435c9a784ea88398560: Status 404 returned error can't find the container with id 74a00d7638b03a191b050a76f3e01ff248ea785ee4ce4435c9a784ea88398560
Jul 09 18:13:43.721775 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.721682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmrmj\" (UniqueName: \"kubernetes.io/projected/3a1e2296-0a12-4284-a1ef-3e8b41a11556-kube-api-access-bmrmj\") pod \"authorino-8df697c9f-lv8th\" (UID: \"3a1e2296-0a12-4284-a1ef-3e8b41a11556\") " pod="kuadrant-system/authorino-8df697c9f-lv8th"
Jul 09 18:13:43.734247 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.734215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmrmj\" (UniqueName: \"kubernetes.io/projected/3a1e2296-0a12-4284-a1ef-3e8b41a11556-kube-api-access-bmrmj\") pod \"authorino-8df697c9f-lv8th\" (UID: \"3a1e2296-0a12-4284-a1ef-3e8b41a11556\") " pod="kuadrant-system/authorino-8df697c9f-lv8th"
Jul 09 18:13:43.865398 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.865355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-b8bml" event={"ID":"25725ad9-41b6-4c06-af4e-48d4f37cbc60","Type":"ContainerStarted","Data":"74a00d7638b03a191b050a76f3e01ff248ea785ee4ce4435c9a784ea88398560"}
Jul 09 18:13:43.912291 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:43.912247    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-lv8th"
Jul 09 18:13:44.064333 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:44.064304    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-lv8th"]
Jul 09 18:13:44.066110 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:13:44.066075    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a1e2296_0a12_4284_a1ef_3e8b41a11556.slice/crio-85724806c0fa98b6844b8019b82d0f28b731106728029cfb245faaa60adc576c WatchSource:0}: Error finding container 85724806c0fa98b6844b8019b82d0f28b731106728029cfb245faaa60adc576c: Status 404 returned error can't find the container with id 85724806c0fa98b6844b8019b82d0f28b731106728029cfb245faaa60adc576c
Jul 09 18:13:44.873937 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:44.873772    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-lv8th" event={"ID":"3a1e2296-0a12-4284-a1ef-3e8b41a11556","Type":"ContainerStarted","Data":"85724806c0fa98b6844b8019b82d0f28b731106728029cfb245faaa60adc576c"}
Jul 09 18:13:45.404677 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:45.404637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:47.887647 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:47.887599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-b8bml" event={"ID":"25725ad9-41b6-4c06-af4e-48d4f37cbc60","Type":"ContainerStarted","Data":"4477b77e1f636138f4561882a9d7b346844b77b2939cced2aa7394e2b2644b90"}
Jul 09 18:13:47.888983 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:47.888956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-lv8th" event={"ID":"3a1e2296-0a12-4284-a1ef-3e8b41a11556","Type":"ContainerStarted","Data":"927d973ce39863deb922fb5e856d8abe3572d069e749e857b119dc9a0efe10a3"}
Jul 09 18:13:47.935083 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:47.935031    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c89bb85-b8bml" podStartSLOduration=1.728286889 podStartE2EDuration="4.935012322s" totalImagesPullingTime="3.206725433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:47.911211668 +0000 UTC m=+551.098189448" watchObservedRunningTime="2026-07-09 18:13:47.935012322 +0000 UTC m=+551.121990090"
Jul 09 18:13:47.935246 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:47.935225    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-lv8th" podStartSLOduration=2.136792379 podStartE2EDuration="4.935219777s" totalImagesPullingTime="2.798427398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:47.934324285 +0000 UTC m=+551.121302054" watchObservedRunningTime="2026-07-09 18:13:47.935219777 +0000 UTC m=+551.122197545"
Jul 09 18:13:47.974325 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:47.974291    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-b8bml"]
Jul 09 18:13:49.898732 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:49.898663    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c89bb85-b8bml" podUID="25725ad9-41b6-4c06-af4e-48d4f37cbc60" containerName="authorino" containerID="cri-o://4477b77e1f636138f4561882a9d7b346844b77b2939cced2aa7394e2b2644b90" gracePeriod=30
Jul 09 18:13:50.152492 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:50.152422    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-b8bml"
Jul 09 18:13:50.178203 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:50.178167    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fls8\" (UniqueName: \"kubernetes.io/projected/25725ad9-41b6-4c06-af4e-48d4f37cbc60-kube-api-access-4fls8\") pod \"25725ad9-41b6-4c06-af4e-48d4f37cbc60\" (UID: \"25725ad9-41b6-4c06-af4e-48d4f37cbc60\") "
Jul 09 18:13:50.180445 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:50.180404    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25725ad9-41b6-4c06-af4e-48d4f37cbc60-kube-api-access-4fls8" (OuterVolumeSpecName: "kube-api-access-4fls8") pod "25725ad9-41b6-4c06-af4e-48d4f37cbc60" (UID: "25725ad9-41b6-4c06-af4e-48d4f37cbc60"). InnerVolumeSpecName "kube-api-access-4fls8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:13:50.279178 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:50.279140    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fls8\" (UniqueName: \"kubernetes.io/projected/25725ad9-41b6-4c06-af4e-48d4f37cbc60-kube-api-access-4fls8\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:13:50.904095 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:50.904056    2578 generic.go:356] "Generic (PLEG): container finished" podID="25725ad9-41b6-4c06-af4e-48d4f37cbc60" containerID="4477b77e1f636138f4561882a9d7b346844b77b2939cced2aa7394e2b2644b90" exitCode=0
Jul 09 18:13:50.904095 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:50.904096    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-b8bml"
Jul 09 18:13:50.904607 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:50.904141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-b8bml" event={"ID":"25725ad9-41b6-4c06-af4e-48d4f37cbc60","Type":"ContainerDied","Data":"4477b77e1f636138f4561882a9d7b346844b77b2939cced2aa7394e2b2644b90"}
Jul 09 18:13:50.904607 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:50.904176    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-b8bml" event={"ID":"25725ad9-41b6-4c06-af4e-48d4f37cbc60","Type":"ContainerDied","Data":"74a00d7638b03a191b050a76f3e01ff248ea785ee4ce4435c9a784ea88398560"}
Jul 09 18:13:50.904607 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:50.904198    2578 scope.go:117] "RemoveContainer" containerID="4477b77e1f636138f4561882a9d7b346844b77b2939cced2aa7394e2b2644b90"
Jul 09 18:13:50.912523 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:50.912500    2578 scope.go:117] "RemoveContainer" containerID="4477b77e1f636138f4561882a9d7b346844b77b2939cced2aa7394e2b2644b90"
Jul 09 18:13:50.912833 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:13:50.912786    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4477b77e1f636138f4561882a9d7b346844b77b2939cced2aa7394e2b2644b90\": container with ID starting with 4477b77e1f636138f4561882a9d7b346844b77b2939cced2aa7394e2b2644b90 not found: ID does not exist" containerID="4477b77e1f636138f4561882a9d7b346844b77b2939cced2aa7394e2b2644b90"
Jul 09 18:13:50.912903 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:50.912844    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4477b77e1f636138f4561882a9d7b346844b77b2939cced2aa7394e2b2644b90"} err="failed to get container status \"4477b77e1f636138f4561882a9d7b346844b77b2939cced2aa7394e2b2644b90\": rpc error: code = NotFound desc = could not find container \"4477b77e1f636138f4561882a9d7b346844b77b2939cced2aa7394e2b2644b90\": container with ID starting with 4477b77e1f636138f4561882a9d7b346844b77b2939cced2aa7394e2b2644b90 not found: ID does not exist"
Jul 09 18:13:50.926242 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:50.926206    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-b8bml"]
Jul 09 18:13:50.931859 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:50.931831    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-b8bml"]
Jul 09 18:13:51.403057 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:13:51.403027    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25725ad9-41b6-4c06-af4e-48d4f37cbc60" path="/var/lib/kubelet/pods/25725ad9-41b6-4c06-af4e-48d4f37cbc60/volumes"
Jul 09 18:14:03.988394 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:03.988354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:14:04.964141 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:04.964098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:14:05.463000 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:05.462974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:14:10.238491 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.238450    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-87vr8"]
Jul 09 18:14:10.238989 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.238922    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25725ad9-41b6-4c06-af4e-48d4f37cbc60" containerName="authorino"
Jul 09 18:14:10.238989 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.238940    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="25725ad9-41b6-4c06-af4e-48d4f37cbc60" containerName="authorino"
Jul 09 18:14:10.239126 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.239031    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="25725ad9-41b6-4c06-af4e-48d4f37cbc60" containerName="authorino"
Jul 09 18:14:10.244812 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.244764    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-87vr8"
Jul 09 18:14:10.248166 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.248139    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 09 18:14:10.249676 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.249655    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-87vr8"]
Jul 09 18:14:10.348582 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.348547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c3b6827-03a1-4b78-867f-a31dd2fbbf64-tls-cert\") pod \"authorino-7c77957575-87vr8\" (UID: \"2c3b6827-03a1-4b78-867f-a31dd2fbbf64\") " pod="kuadrant-system/authorino-7c77957575-87vr8"
Jul 09 18:14:10.348582 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.348584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvjd4\" (UniqueName: \"kubernetes.io/projected/2c3b6827-03a1-4b78-867f-a31dd2fbbf64-kube-api-access-zvjd4\") pod \"authorino-7c77957575-87vr8\" (UID: \"2c3b6827-03a1-4b78-867f-a31dd2fbbf64\") " pod="kuadrant-system/authorino-7c77957575-87vr8"
Jul 09 18:14:10.449449 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.449412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c3b6827-03a1-4b78-867f-a31dd2fbbf64-tls-cert\") pod \"authorino-7c77957575-87vr8\" (UID: \"2c3b6827-03a1-4b78-867f-a31dd2fbbf64\") " pod="kuadrant-system/authorino-7c77957575-87vr8"
Jul 09 18:14:10.449449 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.449457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvjd4\" (UniqueName: \"kubernetes.io/projected/2c3b6827-03a1-4b78-867f-a31dd2fbbf64-kube-api-access-zvjd4\") pod \"authorino-7c77957575-87vr8\" (UID: \"2c3b6827-03a1-4b78-867f-a31dd2fbbf64\") " pod="kuadrant-system/authorino-7c77957575-87vr8"
Jul 09 18:14:10.452009 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.451989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c3b6827-03a1-4b78-867f-a31dd2fbbf64-tls-cert\") pod \"authorino-7c77957575-87vr8\" (UID: \"2c3b6827-03a1-4b78-867f-a31dd2fbbf64\") " pod="kuadrant-system/authorino-7c77957575-87vr8"
Jul 09 18:14:10.458775 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.458740    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvjd4\" (UniqueName: \"kubernetes.io/projected/2c3b6827-03a1-4b78-867f-a31dd2fbbf64-kube-api-access-zvjd4\") pod \"authorino-7c77957575-87vr8\" (UID: \"2c3b6827-03a1-4b78-867f-a31dd2fbbf64\") " pod="kuadrant-system/authorino-7c77957575-87vr8"
Jul 09 18:14:10.554942 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.554850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-87vr8"
Jul 09 18:14:10.696068 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.696038    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-87vr8"]
Jul 09 18:14:10.698708 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:14:10.698659    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c3b6827_03a1_4b78_867f_a31dd2fbbf64.slice/crio-c78335ad5120ea7b2830b37cf2d308fad0ebf486f9fdf3f06667b072a0ad7fdc WatchSource:0}: Error finding container c78335ad5120ea7b2830b37cf2d308fad0ebf486f9fdf3f06667b072a0ad7fdc: Status 404 returned error can't find the container with id c78335ad5120ea7b2830b37cf2d308fad0ebf486f9fdf3f06667b072a0ad7fdc
Jul 09 18:14:10.997177 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:10.997144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-87vr8" event={"ID":"2c3b6827-03a1-4b78-867f-a31dd2fbbf64","Type":"ContainerStarted","Data":"c78335ad5120ea7b2830b37cf2d308fad0ebf486f9fdf3f06667b072a0ad7fdc"}
Jul 09 18:14:12.003938 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:12.003900    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-87vr8" event={"ID":"2c3b6827-03a1-4b78-867f-a31dd2fbbf64","Type":"ContainerStarted","Data":"67977f9fb7afb482e864844603951d3c71ecf368be98a1d1643e25b866c73dcf"}
Jul 09 18:14:12.022050 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:12.021990    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-87vr8" podStartSLOduration=1.587436969 podStartE2EDuration="2.021970785s" totalImagesPullingTime="434.533816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:14:12.020302402 +0000 UTC m=+575.207280170" watchObservedRunningTime="2026-07-09 18:14:12.021970785 +0000 UTC m=+575.208948552"
Jul 09 18:14:12.052857 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:12.052824    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-lv8th"]
Jul 09 18:14:12.053050 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:12.053024    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-lv8th" podUID="3a1e2296-0a12-4284-a1ef-3e8b41a11556" containerName="authorino" containerID="cri-o://927d973ce39863deb922fb5e856d8abe3572d069e749e857b119dc9a0efe10a3" gracePeriod=30
Jul 09 18:14:12.311024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:12.310996    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-lv8th"
Jul 09 18:14:12.468044 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:12.467997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmrmj\" (UniqueName: \"kubernetes.io/projected/3a1e2296-0a12-4284-a1ef-3e8b41a11556-kube-api-access-bmrmj\") pod \"3a1e2296-0a12-4284-a1ef-3e8b41a11556\" (UID: \"3a1e2296-0a12-4284-a1ef-3e8b41a11556\") "
Jul 09 18:14:12.470165 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:12.470123    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1e2296-0a12-4284-a1ef-3e8b41a11556-kube-api-access-bmrmj" (OuterVolumeSpecName: "kube-api-access-bmrmj") pod "3a1e2296-0a12-4284-a1ef-3e8b41a11556" (UID: "3a1e2296-0a12-4284-a1ef-3e8b41a11556"). InnerVolumeSpecName "kube-api-access-bmrmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:14:12.569280 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:12.569179    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmrmj\" (UniqueName: \"kubernetes.io/projected/3a1e2296-0a12-4284-a1ef-3e8b41a11556-kube-api-access-bmrmj\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:14:13.011471 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:13.011434    2578 generic.go:356] "Generic (PLEG): container finished" podID="3a1e2296-0a12-4284-a1ef-3e8b41a11556" containerID="927d973ce39863deb922fb5e856d8abe3572d069e749e857b119dc9a0efe10a3" exitCode=0
Jul 09 18:14:13.011959 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:13.011481    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-lv8th"
Jul 09 18:14:13.011959 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:13.011520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-lv8th" event={"ID":"3a1e2296-0a12-4284-a1ef-3e8b41a11556","Type":"ContainerDied","Data":"927d973ce39863deb922fb5e856d8abe3572d069e749e857b119dc9a0efe10a3"}
Jul 09 18:14:13.011959 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:13.011563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-lv8th" event={"ID":"3a1e2296-0a12-4284-a1ef-3e8b41a11556","Type":"ContainerDied","Data":"85724806c0fa98b6844b8019b82d0f28b731106728029cfb245faaa60adc576c"}
Jul 09 18:14:13.011959 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:13.011581    2578 scope.go:117] "RemoveContainer" containerID="927d973ce39863deb922fb5e856d8abe3572d069e749e857b119dc9a0efe10a3"
Jul 09 18:14:13.020005 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:13.019986    2578 scope.go:117] "RemoveContainer" containerID="927d973ce39863deb922fb5e856d8abe3572d069e749e857b119dc9a0efe10a3"
Jul 09 18:14:13.020285 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:14:13.020266    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927d973ce39863deb922fb5e856d8abe3572d069e749e857b119dc9a0efe10a3\": container with ID starting with 927d973ce39863deb922fb5e856d8abe3572d069e749e857b119dc9a0efe10a3 not found: ID does not exist" containerID="927d973ce39863deb922fb5e856d8abe3572d069e749e857b119dc9a0efe10a3"
Jul 09 18:14:13.020319 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:13.020295    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927d973ce39863deb922fb5e856d8abe3572d069e749e857b119dc9a0efe10a3"} err="failed to get container status \"927d973ce39863deb922fb5e856d8abe3572d069e749e857b119dc9a0efe10a3\": rpc error: code = NotFound desc = could not find container \"927d973ce39863deb922fb5e856d8abe3572d069e749e857b119dc9a0efe10a3\": container with ID starting with 927d973ce39863deb922fb5e856d8abe3572d069e749e857b119dc9a0efe10a3 not found: ID does not exist"
Jul 09 18:14:13.040569 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:13.040533    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-lv8th"]
Jul 09 18:14:13.044300 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:13.044266    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-lv8th"]
Jul 09 18:14:13.402846 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:13.402750    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a1e2296-0a12-4284-a1ef-3e8b41a11556" path="/var/lib/kubelet/pods/3a1e2296-0a12-4284-a1ef-3e8b41a11556/volumes"
Jul 09 18:14:37.241243 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:37.241212    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:14:37.241759 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:37.241739    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:14:37.251143 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:37.251115    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:14:37.251908 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:37.251490    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:14:49.959680 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:49.959641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:14:50.042341 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:50.042308    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:14:50.299448 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:50.299347    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:14:50.397306 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:50.397273    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:14:50.780809 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:50.780771    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:14:50.880293 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:14:50.880260    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:15.517115 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:15:15.517062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:59.232176 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:15:59.232136    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:00.733402 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:00.733372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:26.093585 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:26.093543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:26.415041 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:26.414945    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:38.741813 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:38.741761    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:38.843256 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:38.843220    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:42.903112 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:42.903070    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:44.209996 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:44.209958    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:45.434078 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.434039    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"]
Jul 09 18:16:45.434494 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.434345    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a1e2296-0a12-4284-a1ef-3e8b41a11556" containerName="authorino"
Jul 09 18:16:45.434494 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.434356    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a1e2296-0a12-4284-a1ef-3e8b41a11556" containerName="authorino"
Jul 09 18:16:45.434494 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.434431    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a1e2296-0a12-4284-a1ef-3e8b41a11556" containerName="authorino"
Jul 09 18:16:45.437490 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.437472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.439989 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.439963    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:16:45.440130 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.439962    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:16:45.440130 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.440005    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 09 18:16:45.440130 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.440082    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-7sv49\""
Jul 09 18:16:45.448297 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.448274    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"]
Jul 09 18:16:45.593841 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.593761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.594035 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.593868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.594035 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.593928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.594035 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.593965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4fz9\" (UniqueName: \"kubernetes.io/projected/778f897b-c5cf-431c-a235-dba2d4745bb4-kube-api-access-r4fz9\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.594035 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.593984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.594035 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.594007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.594196 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.594066    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.594196 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.594112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.594196 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.594135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778f897b-c5cf-431c-a235-dba2d4745bb4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.695599 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.695511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.695599 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.695553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.695599 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.695581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.695861 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.695702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4fz9\" (UniqueName: \"kubernetes.io/projected/778f897b-c5cf-431c-a235-dba2d4745bb4-kube-api-access-r4fz9\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.695861 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.695744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.695861 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.695775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.695861 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.695849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.696058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.695883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.696058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.695909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778f897b-c5cf-431c-a235-dba2d4745bb4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.696058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.695984    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.696213 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.696082    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.696269 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.696243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.696321 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.696274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.696625 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.696603    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778f897b-c5cf-431c-a235-dba2d4745bb4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.697975 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.697958    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.698308 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.698291    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.714892 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.714858    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.715462 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.715442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4fz9\" (UniqueName: \"kubernetes.io/projected/778f897b-c5cf-431c-a235-dba2d4745bb4-kube-api-access-r4fz9\") pod \"router-gateway-1-openshift-default-799f46c59b-fwgns\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.748555 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.748517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:45.889524 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.889495    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"]
Jul 09 18:16:45.891933 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:16:45.891904    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778f897b_c5cf_431c_a235_dba2d4745bb4.slice/crio-1be811b971dbd4a998dbbaea8f58057a95d754667f0e5b2fd30223e3457c2dfd WatchSource:0}: Error finding container 1be811b971dbd4a998dbbaea8f58057a95d754667f0e5b2fd30223e3457c2dfd: Status 404 returned error can't find the container with id 1be811b971dbd4a998dbbaea8f58057a95d754667f0e5b2fd30223e3457c2dfd
Jul 09 18:16:45.893966 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:45.893947    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:16:46.633177 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:46.633135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" event={"ID":"778f897b-c5cf-431c-a235-dba2d4745bb4","Type":"ContainerStarted","Data":"1be811b971dbd4a998dbbaea8f58057a95d754667f0e5b2fd30223e3457c2dfd"}
Jul 09 18:16:48.219847 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:48.219787    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:16:48.220218 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:48.219923    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:16:48.220218 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:48.219980    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:16:48.643359 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:48.643255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" event={"ID":"778f897b-c5cf-431c-a235-dba2d4745bb4","Type":"ContainerStarted","Data":"5ef3118bee22dc2a8e85ed6b1e70d0901ce4e0d4c96b33d4a0fcd4e74b7e56f3"}
Jul 09 18:16:48.671847 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:48.671760    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podStartSLOduration=1.34621986 podStartE2EDuration="3.671738576s" totalImagesPullingTime="2.325518716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:16:48.670233062 +0000 UTC m=+731.857210830" watchObservedRunningTime="2026-07-09 18:16:48.671738576 +0000 UTC m=+731.858716344"
Jul 09 18:16:48.748917 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:48.748876    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:50.790020 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:50.789961    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 18:16:51.816199 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:51.816142    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 18:16:52.816640 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:52.816586    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 18:16:53.827923 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:53.827013    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 18:16:54.868007 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:54.867957    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 18:16:55.909090 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:55.909027    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 18:16:56.952036 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:56.951989    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 18:16:57.691443 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:57.691326    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": read tcp 10.132.0.2:48948->10.132.0.29:15021: read: connection reset by peer"
Jul 09 18:16:57.749242 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:57.749195    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Jul 09 18:16:58.684537 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:58.684505    2578 generic.go:356] "Generic (PLEG): container finished" podID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerID="5ef3118bee22dc2a8e85ed6b1e70d0901ce4e0d4c96b33d4a0fcd4e74b7e56f3" exitCode=0
Jul 09 18:16:58.684940 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:58.684585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" event={"ID":"778f897b-c5cf-431c-a235-dba2d4745bb4","Type":"ContainerDied","Data":"5ef3118bee22dc2a8e85ed6b1e70d0901ce4e0d4c96b33d4a0fcd4e74b7e56f3"}
Jul 09 18:16:58.685023 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:58.684995    2578 scope.go:117] "RemoveContainer" containerID="5ef3118bee22dc2a8e85ed6b1e70d0901ce4e0d4c96b33d4a0fcd4e74b7e56f3"
Jul 09 18:16:58.686927 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:58.686897    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:16:58.687079 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:58.686979    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:16:58.687079 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:58.687028    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:16:58.748709 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:58.748685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:59.690543 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:59.690505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" event={"ID":"778f897b-c5cf-431c-a235-dba2d4745bb4","Type":"ContainerStarted","Data":"ce89618138985a3ddf1c211e511777e9979327aaa2e2d84767320b16c646321f"}
Jul 09 18:16:59.748834 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:59.748769    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:16:59.749071 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:16:59.749041    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Jul 09 18:17:00.749569 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:00.749524    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:17:00.749998 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:00.749824    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Jul 09 18:17:01.749552 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:01.749515    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Jul 09 18:17:02.749490 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:02.749452    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Jul 09 18:17:03.749453 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:03.749405    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Jul 09 18:17:04.774321 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:04.774290    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:17:04.775697 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:04.775674    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:17:12.881475 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:12.881436    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"]
Jul 09 18:17:12.886453 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:12.886421    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:12.889434 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:12.889409    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:17:12.889586 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:12.889416    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 09 18:17:12.889586 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:12.889462    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-s8z92\""
Jul 09 18:17:12.899734 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:12.899701    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"]
Jul 09 18:17:12.901751 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:12.901720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:12.943103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:12.943056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-home\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:12.943103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:12.943100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-dshm\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:12.943325 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:12.943124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:12.943325 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:12.943190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:12.943325 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:12.943265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6fb84a-eee2-403b-874a-b211c61c21b4-tls-certs\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:12.943325 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:12.943302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wf6\" (UniqueName: \"kubernetes.io/projected/3c6fb84a-eee2-403b-874a-b211c61c21b4-kube-api-access-q5wf6\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:12.943447 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:12.943353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-model-cache\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.044176 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.044133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-home\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.044176 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.044173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-dshm\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.044428 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.044198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.044428 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.044217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.044428 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.044246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6fb84a-eee2-403b-874a-b211c61c21b4-tls-certs\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.044428 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.044263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wf6\" (UniqueName: \"kubernetes.io/projected/3c6fb84a-eee2-403b-874a-b211c61c21b4-kube-api-access-q5wf6\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.044428 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.044302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-model-cache\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.044692 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.044598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-home\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.044692 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.044645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.044836 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.044791    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.044896 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.044846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-model-cache\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.046543 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.046518    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-dshm\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.046742 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.046724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6fb84a-eee2-403b-874a-b211c61c21b4-tls-certs\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.052994 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.052968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wf6\" (UniqueName: \"kubernetes.io/projected/3c6fb84a-eee2-403b-874a-b211c61c21b4-kube-api-access-q5wf6\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-wzlgk\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.197442 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.197410    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:13.305371 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.305329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:13.350179 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.350151    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"]
Jul 09 18:17:13.352821 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:17:13.352778    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6fb84a_eee2_403b_874a_b211c61c21b4.slice/crio-1b79774968b35816a0eeb3175c6dabf38eb3255053783644b82787383fa9a468 WatchSource:0}: Error finding container 1b79774968b35816a0eeb3175c6dabf38eb3255053783644b82787383fa9a468: Status 404 returned error can't find the container with id 1b79774968b35816a0eeb3175c6dabf38eb3255053783644b82787383fa9a468
Jul 09 18:17:13.751703 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.751665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" event={"ID":"3c6fb84a-eee2-403b-874a-b211c61c21b4","Type":"ContainerStarted","Data":"1b79774968b35816a0eeb3175c6dabf38eb3255053783644b82787383fa9a468"}
Jul 09 18:17:13.988961 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:13.988926    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:17.433680 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:17.433648    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:18.779736 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:18.779700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" event={"ID":"3c6fb84a-eee2-403b-874a-b211c61c21b4","Type":"ContainerStarted","Data":"7368a1465935bd0dbaf07164afaecf94537c779c1fe704a58c5538fcebfbf4dd"}
Jul 09 18:17:21.776997 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:21.776967    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:21.800045 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:21.800013    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerID="7368a1465935bd0dbaf07164afaecf94537c779c1fe704a58c5538fcebfbf4dd" exitCode=0
Jul 09 18:17:21.800211 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:21.800062    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" event={"ID":"3c6fb84a-eee2-403b-874a-b211c61c21b4","Type":"ContainerDied","Data":"7368a1465935bd0dbaf07164afaecf94537c779c1fe704a58c5538fcebfbf4dd"}
Jul 09 18:17:23.811676 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:23.811637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" event={"ID":"3c6fb84a-eee2-403b-874a-b211c61c21b4","Type":"ContainerStarted","Data":"12e271170cc3db897cd0ba3f331bb3ed057f683a8858defaaa497edaee2aa930"}
Jul 09 18:17:23.832807 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:23.832735    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" podStartSLOduration=2.749474815 podStartE2EDuration="11.832717003s" totalImagesPullingTime="5.645519544s" totalInitContainerRuntime="3.437722644s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:23.831014701 +0000 UTC m=+767.017992470" watchObservedRunningTime="2026-07-09 18:17:23.832717003 +0000 UTC m=+767.019694770"
Jul 09 18:17:24.092970 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.092888    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"]
Jul 09 18:17:24.096847 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.096829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.106122 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.106095    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"]
Jul 09 18:17:24.147305 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.147266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.147487 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.147323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.147487 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.147342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrmsj\" (UniqueName: \"kubernetes.io/projected/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-kube-api-access-rrmsj\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.147487 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.147372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.147487 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.147452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.147487 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.147480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.147652 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.147505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.147652 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.147561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.147652 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.147594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.198322 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.198287    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:24.248890 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.248857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.249080 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.248905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.249080 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.248930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.249080 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.248950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.249080 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.248974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.249080 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.249033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.249080 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.249061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.249080 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.249076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmsj\" (UniqueName: \"kubernetes.io/projected/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-kube-api-access-rrmsj\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.249489 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.249119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.249489 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.249189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.249489 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.249298    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.249604 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.249513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.249604 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.249547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.249944 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.249925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.251352 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.251316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.251584 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.251563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.259145 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.259118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.259358 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.259332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrmsj\" (UniqueName: \"kubernetes.io/projected/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-kube-api-access-rrmsj\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tx2cs\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.406913 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.406789    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:24.545272 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.545247    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"]
Jul 09 18:17:24.547188 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:17:24.547161    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ddb135e_1993_48c8_9b70_ed8f86d6bc8d.slice/crio-cd36b5d06808791d0f1e94337360a6533769826af841cff2fdc64ff821480457 WatchSource:0}: Error finding container cd36b5d06808791d0f1e94337360a6533769826af841cff2fdc64ff821480457: Status 404 returned error can't find the container with id cd36b5d06808791d0f1e94337360a6533769826af841cff2fdc64ff821480457
Jul 09 18:17:24.549431 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.549395    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:17:24.549574 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.549476    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:17:24.549574 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.549516    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:17:24.819036 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.818994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs" event={"ID":"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d","Type":"ContainerStarted","Data":"e03ddd03cf6f20377a900c84467f0dcffa72bd772b8d6c968c7587e17584330e"}
Jul 09 18:17:24.819449 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.819061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs" event={"ID":"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d","Type":"ContainerStarted","Data":"cd36b5d06808791d0f1e94337360a6533769826af841cff2fdc64ff821480457"}
Jul 09 18:17:24.841702 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:24.841637    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs" podStartSLOduration=0.84161951 podStartE2EDuration="841.61951ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:24.841264803 +0000 UTC m=+768.028242596" watchObservedRunningTime="2026-07-09 18:17:24.84161951 +0000 UTC m=+768.028597277"
Jul 09 18:17:25.406996 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:25.406959    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:25.412337 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:25.412311    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:25.824854 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:25.824789    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:25.825911 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:25.825893    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:17:25.879281 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:25.879256    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"]
Jul 09 18:17:25.879491 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:25.879466    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy" containerID="cri-o://ce89618138985a3ddf1c211e511777e9979327aaa2e2d84767320b16c646321f" gracePeriod=30
Jul 09 18:17:31.164058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.163973    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:17:31.316266 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.316230    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-podinfo\") pod \"778f897b-c5cf-431c-a235-dba2d4745bb4\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") "
Jul 09 18:17:31.316467 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.316276    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4fz9\" (UniqueName: \"kubernetes.io/projected/778f897b-c5cf-431c-a235-dba2d4745bb4-kube-api-access-r4fz9\") pod \"778f897b-c5cf-431c-a235-dba2d4745bb4\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") "
Jul 09 18:17:31.316467 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.316309    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-data\") pod \"778f897b-c5cf-431c-a235-dba2d4745bb4\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") "
Jul 09 18:17:31.316467 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.316324    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-credential-socket\") pod \"778f897b-c5cf-431c-a235-dba2d4745bb4\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") "
Jul 09 18:17:31.316467 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.316393    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-token\") pod \"778f897b-c5cf-431c-a235-dba2d4745bb4\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") "
Jul 09 18:17:31.316467 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.316453    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-workload-socket\") pod \"778f897b-c5cf-431c-a235-dba2d4745bb4\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") "
Jul 09 18:17:31.316727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.316477    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-envoy\") pod \"778f897b-c5cf-431c-a235-dba2d4745bb4\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") "
Jul 09 18:17:31.316727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.316506    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-workload-certs\") pod \"778f897b-c5cf-431c-a235-dba2d4745bb4\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") "
Jul 09 18:17:31.316727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.316531    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778f897b-c5cf-431c-a235-dba2d4745bb4-istiod-ca-cert\") pod \"778f897b-c5cf-431c-a235-dba2d4745bb4\" (UID: \"778f897b-c5cf-431c-a235-dba2d4745bb4\") "
Jul 09 18:17:31.316727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.316668    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "778f897b-c5cf-431c-a235-dba2d4745bb4" (UID: "778f897b-c5cf-431c-a235-dba2d4745bb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:31.316727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.316683    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-data" (OuterVolumeSpecName: "istio-data") pod "778f897b-c5cf-431c-a235-dba2d4745bb4" (UID: "778f897b-c5cf-431c-a235-dba2d4745bb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:31.316990 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.316831    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-data\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:17:31.316990 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.316853    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-credential-socket\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:17:31.316990 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.316847    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "778f897b-c5cf-431c-a235-dba2d4745bb4" (UID: "778f897b-c5cf-431c-a235-dba2d4745bb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:31.317098 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.317013    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "778f897b-c5cf-431c-a235-dba2d4745bb4" (UID: "778f897b-c5cf-431c-a235-dba2d4745bb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:31.317133 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.317105    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778f897b-c5cf-431c-a235-dba2d4745bb4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "778f897b-c5cf-431c-a235-dba2d4745bb4" (UID: "778f897b-c5cf-431c-a235-dba2d4745bb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:17:31.318831 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.318764    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "778f897b-c5cf-431c-a235-dba2d4745bb4" (UID: "778f897b-c5cf-431c-a235-dba2d4745bb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:31.318945 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.318832    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778f897b-c5cf-431c-a235-dba2d4745bb4-kube-api-access-r4fz9" (OuterVolumeSpecName: "kube-api-access-r4fz9") pod "778f897b-c5cf-431c-a235-dba2d4745bb4" (UID: "778f897b-c5cf-431c-a235-dba2d4745bb4"). InnerVolumeSpecName "kube-api-access-r4fz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:17:31.319015 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.318995    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "778f897b-c5cf-431c-a235-dba2d4745bb4" (UID: "778f897b-c5cf-431c-a235-dba2d4745bb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 18:17:31.319061 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.319011    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-token" (OuterVolumeSpecName: "istio-token") pod "778f897b-c5cf-431c-a235-dba2d4745bb4" (UID: "778f897b-c5cf-431c-a235-dba2d4745bb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:17:31.418231 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.418151    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-token\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:17:31.418231 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.418182    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-workload-socket\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:17:31.418231 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.418191    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-envoy\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:17:31.418231 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.418201    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778f897b-c5cf-431c-a235-dba2d4745bb4-workload-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:17:31.418231 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.418209    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778f897b-c5cf-431c-a235-dba2d4745bb4-istiod-ca-cert\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:17:31.418231 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.418217    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778f897b-c5cf-431c-a235-dba2d4745bb4-istio-podinfo\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:17:31.418231 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.418224    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4fz9\" (UniqueName: \"kubernetes.io/projected/778f897b-c5cf-431c-a235-dba2d4745bb4-kube-api-access-r4fz9\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:17:31.854392 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.854353    2578 generic.go:356] "Generic (PLEG): container finished" podID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerID="ce89618138985a3ddf1c211e511777e9979327aaa2e2d84767320b16c646321f" exitCode=0
Jul 09 18:17:31.854596 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.854402    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"
Jul 09 18:17:31.854596 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.854435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" event={"ID":"778f897b-c5cf-431c-a235-dba2d4745bb4","Type":"ContainerDied","Data":"ce89618138985a3ddf1c211e511777e9979327aaa2e2d84767320b16c646321f"}
Jul 09 18:17:31.854596 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.854495    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns" event={"ID":"778f897b-c5cf-431c-a235-dba2d4745bb4","Type":"ContainerDied","Data":"1be811b971dbd4a998dbbaea8f58057a95d754667f0e5b2fd30223e3457c2dfd"}
Jul 09 18:17:31.854596 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.854524    2578 scope.go:117] "RemoveContainer" containerID="ce89618138985a3ddf1c211e511777e9979327aaa2e2d84767320b16c646321f"
Jul 09 18:17:31.867806 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.867775    2578 scope.go:117] "RemoveContainer" containerID="5ef3118bee22dc2a8e85ed6b1e70d0901ce4e0d4c96b33d4a0fcd4e74b7e56f3"
Jul 09 18:17:31.874032 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.874006    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"]
Jul 09 18:17:31.877011 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.876989    2578 scope.go:117] "RemoveContainer" containerID="ce89618138985a3ddf1c211e511777e9979327aaa2e2d84767320b16c646321f"
Jul 09 18:17:31.877380 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:17:31.877359    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce89618138985a3ddf1c211e511777e9979327aaa2e2d84767320b16c646321f\": container with ID starting with ce89618138985a3ddf1c211e511777e9979327aaa2e2d84767320b16c646321f not found: ID does not exist" containerID="ce89618138985a3ddf1c211e511777e9979327aaa2e2d84767320b16c646321f"
Jul 09 18:17:31.877472 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.877394    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce89618138985a3ddf1c211e511777e9979327aaa2e2d84767320b16c646321f"} err="failed to get container status \"ce89618138985a3ddf1c211e511777e9979327aaa2e2d84767320b16c646321f\": rpc error: code = NotFound desc = could not find container \"ce89618138985a3ddf1c211e511777e9979327aaa2e2d84767320b16c646321f\": container with ID starting with ce89618138985a3ddf1c211e511777e9979327aaa2e2d84767320b16c646321f not found: ID does not exist"
Jul 09 18:17:31.877472 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.877413    2578 scope.go:117] "RemoveContainer" containerID="5ef3118bee22dc2a8e85ed6b1e70d0901ce4e0d4c96b33d4a0fcd4e74b7e56f3"
Jul 09 18:17:31.877698 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:17:31.877680    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef3118bee22dc2a8e85ed6b1e70d0901ce4e0d4c96b33d4a0fcd4e74b7e56f3\": container with ID starting with 5ef3118bee22dc2a8e85ed6b1e70d0901ce4e0d4c96b33d4a0fcd4e74b7e56f3 not found: ID does not exist" containerID="5ef3118bee22dc2a8e85ed6b1e70d0901ce4e0d4c96b33d4a0fcd4e74b7e56f3"
Jul 09 18:17:31.877763 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.877704    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef3118bee22dc2a8e85ed6b1e70d0901ce4e0d4c96b33d4a0fcd4e74b7e56f3"} err="failed to get container status \"5ef3118bee22dc2a8e85ed6b1e70d0901ce4e0d4c96b33d4a0fcd4e74b7e56f3\": rpc error: code = NotFound desc = could not find container \"5ef3118bee22dc2a8e85ed6b1e70d0901ce4e0d4c96b33d4a0fcd4e74b7e56f3\": container with ID starting with 5ef3118bee22dc2a8e85ed6b1e70d0901ce4e0d4c96b33d4a0fcd4e74b7e56f3 not found: ID does not exist"
Jul 09 18:17:31.878936 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:31.878914    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns"]
Jul 09 18:17:33.198013 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:33.197964    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:33.210932 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:33.210898    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:33.403943 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:33.403910    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" path="/var/lib/kubelet/pods/778f897b-c5cf-431c-a235-dba2d4745bb4/volumes"
Jul 09 18:17:33.876057 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:33.876025    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:17:47.707690 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:47.707628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:49.341067 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:49.341026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:59.895468 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.895385    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"]
Jul 09 18:17:59.895856 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.895723    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy"
Jul 09 18:17:59.895856 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.895736    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy"
Jul 09 18:17:59.895856 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.895750    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy"
Jul 09 18:17:59.895856 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.895755    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy"
Jul 09 18:17:59.895856 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.895827    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy"
Jul 09 18:17:59.896014 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.895929    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="778f897b-c5cf-431c-a235-dba2d4745bb4" containerName="istio-proxy"
Jul 09 18:17:59.909769 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.909730    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:17:59.911298 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.911263    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"]
Jul 09 18:17:59.912560 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.912542    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 09 18:17:59.916515 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.916488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:59.984976 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.984940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-dshm\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:17:59.985154 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.984992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26943d6c-548d-47ed-aff8-d9884a7947ac-tls-certs\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:17:59.985154 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.985028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-kserve-provision-location\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:17:59.985154 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.985053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-home\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:17:59.985154 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.985068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-model-cache\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:17:59.985154 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.985082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-tmp-dir\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:17:59.985317 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:17:59.985175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8jgw\" (UniqueName: \"kubernetes.io/projected/26943d6c-548d-47ed-aff8-d9884a7947ac-kube-api-access-t8jgw\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.086168 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.086124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-dshm\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.086363 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.086183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26943d6c-548d-47ed-aff8-d9884a7947ac-tls-certs\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.086363 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.086206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-kserve-provision-location\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.086363 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.086238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-home\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.086363 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.086268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-model-cache\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.086363 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.086291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-tmp-dir\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.086363 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.086337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8jgw\" (UniqueName: \"kubernetes.io/projected/26943d6c-548d-47ed-aff8-d9884a7947ac-kube-api-access-t8jgw\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.086761 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.086729    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-kserve-provision-location\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.086912 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.086769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-home\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.086912 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.086840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-model-cache\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.086912 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.086860    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-tmp-dir\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.088456 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.088435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-dshm\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.088838 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.088820    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26943d6c-548d-47ed-aff8-d9884a7947ac-tls-certs\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.095564 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.095519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8jgw\" (UniqueName: \"kubernetes.io/projected/26943d6c-548d-47ed-aff8-d9884a7947ac-kube-api-access-t8jgw\") pod \"auth-enabled-test-kserve-5d7b464f94-8sqkq\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.221084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.221038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:00.263409 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.263299    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"]
Jul 09 18:18:00.266052 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.266029    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:00.268702 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.268682    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-epp-sa-dockercfg-zhznx\""
Jul 09 18:18:00.278930 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.278899    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"]
Jul 09 18:18:00.341632 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.341594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:00.374012 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.373986    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"]
Jul 09 18:18:00.375868 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:18:00.375836    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26943d6c_548d_47ed_aff8_d9884a7947ac.slice/crio-b207aabf3f9130a6731cdbbff721a7184a60564720fb043118466e984d153edd WatchSource:0}: Error finding container b207aabf3f9130a6731cdbbff721a7184a60564720fb043118466e984d153edd: Status 404 returned error can't find the container with id b207aabf3f9130a6731cdbbff721a7184a60564720fb043118466e984d153edd
Jul 09 18:18:00.388192 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.388158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ddf9ed-4010-43ec-9b35-5ad76988e291-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:00.388354 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.388211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6zvr\" (UniqueName: \"kubernetes.io/projected/89ddf9ed-4010-43ec-9b35-5ad76988e291-kube-api-access-r6zvr\") pod \"auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:00.388354 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.388271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/89ddf9ed-4010-43ec-9b35-5ad76988e291-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:00.388542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.388349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89ddf9ed-4010-43ec-9b35-5ad76988e291-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:00.489765 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.489663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ddf9ed-4010-43ec-9b35-5ad76988e291-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:00.489765 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.489714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zvr\" (UniqueName: \"kubernetes.io/projected/89ddf9ed-4010-43ec-9b35-5ad76988e291-kube-api-access-r6zvr\") pod \"auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:00.489765 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.489753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/89ddf9ed-4010-43ec-9b35-5ad76988e291-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:00.490068 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.489854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89ddf9ed-4010-43ec-9b35-5ad76988e291-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:00.490254 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.490228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89ddf9ed-4010-43ec-9b35-5ad76988e291-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:00.490325 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.490269    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/89ddf9ed-4010-43ec-9b35-5ad76988e291-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:00.492091 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.492074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ddf9ed-4010-43ec-9b35-5ad76988e291-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:00.498020 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.497997    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6zvr\" (UniqueName: \"kubernetes.io/projected/89ddf9ed-4010-43ec-9b35-5ad76988e291-kube-api-access-r6zvr\") pod \"auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:00.578199 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.578163    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:00.726223 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.726199    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"]
Jul 09 18:18:00.729314 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:18:00.729286    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ddf9ed_4010_43ec_9b35_5ad76988e291.slice/crio-40e1e257fc47026b1f89d5b038017b9b07bfb4bc9b392c7582cfeae56ce2acb2 WatchSource:0}: Error finding container 40e1e257fc47026b1f89d5b038017b9b07bfb4bc9b392c7582cfeae56ce2acb2: Status 404 returned error can't find the container with id 40e1e257fc47026b1f89d5b038017b9b07bfb4bc9b392c7582cfeae56ce2acb2
Jul 09 18:18:00.986007 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.985959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" event={"ID":"26943d6c-548d-47ed-aff8-d9884a7947ac","Type":"ContainerStarted","Data":"a503afafc82c79b8f22dab313ef6d072d67edf5ad79c736349ebaec7fa7dc1af"}
Jul 09 18:18:00.986469 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.986072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" event={"ID":"26943d6c-548d-47ed-aff8-d9884a7947ac","Type":"ContainerStarted","Data":"b207aabf3f9130a6731cdbbff721a7184a60564720fb043118466e984d153edd"}
Jul 09 18:18:00.987729 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.987695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67" event={"ID":"89ddf9ed-4010-43ec-9b35-5ad76988e291","Type":"ContainerStarted","Data":"d9bfaf1fe8c269a484a7314ff6cdc932662f27077200e59a5ab86430463a0290"}
Jul 09 18:18:00.987861 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:00.987744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67" event={"ID":"89ddf9ed-4010-43ec-9b35-5ad76988e291","Type":"ContainerStarted","Data":"40e1e257fc47026b1f89d5b038017b9b07bfb4bc9b392c7582cfeae56ce2acb2"}
Jul 09 18:18:01.994412 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:01.994371    2578 generic.go:356] "Generic (PLEG): container finished" podID="89ddf9ed-4010-43ec-9b35-5ad76988e291" containerID="d9bfaf1fe8c269a484a7314ff6cdc932662f27077200e59a5ab86430463a0290" exitCode=0
Jul 09 18:18:01.995071 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:01.994460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67" event={"ID":"89ddf9ed-4010-43ec-9b35-5ad76988e291","Type":"ContainerDied","Data":"d9bfaf1fe8c269a484a7314ff6cdc932662f27077200e59a5ab86430463a0290"}
Jul 09 18:18:04.009557 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:04.009522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67" event={"ID":"89ddf9ed-4010-43ec-9b35-5ad76988e291","Type":"ContainerStarted","Data":"4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175"}
Jul 09 18:18:04.010022 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:04.009747    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:04.033416 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:04.033354    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67" podStartSLOduration=1.89128958 podStartE2EDuration="4.033335786s" totalImagesPullingTime="1.362782794s" totalInitContainerRuntime="779.263412ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:04.032054442 +0000 UTC m=+807.219032237" watchObservedRunningTime="2026-07-09 18:18:04.033335786 +0000 UTC m=+807.220313553"
Jul 09 18:18:05.015417 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:05.015384    2578 generic.go:356] "Generic (PLEG): container finished" podID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerID="a503afafc82c79b8f22dab313ef6d072d67edf5ad79c736349ebaec7fa7dc1af" exitCode=0
Jul 09 18:18:05.015877 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:05.015464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" event={"ID":"26943d6c-548d-47ed-aff8-d9884a7947ac","Type":"ContainerDied","Data":"a503afafc82c79b8f22dab313ef6d072d67edf5ad79c736349ebaec7fa7dc1af"}
Jul 09 18:18:19.399954 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:19.399914    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:19.961443 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:19.961404    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:25.758340 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:25.758051    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:25.759543 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:25.759512    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"]
Jul 09 18:18:25.829136 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:25.829102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:35.017836 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:35.017781    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:18:40.787110 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:40.787058    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main" containerID="cri-o://12e271170cc3db897cd0ba3f331bb3ed057f683a8858defaaa497edaee2aa930" gracePeriod=45
Jul 09 18:18:40.865735 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:40.865690    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Jul 09 18:18:41.865857 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:41.865761    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Jul 09 18:18:41.866306 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:41.865931    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:18:42.865303 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:42.865254    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Jul 09 18:18:43.865594 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:43.865546    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Jul 09 18:18:44.356251 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:44.356204    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:44.865316 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:44.865271    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Jul 09 18:18:45.865643 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:45.865590    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Jul 09 18:18:46.865276 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:46.865227    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Jul 09 18:18:47.865528 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:47.865482    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Jul 09 18:18:48.865261 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:48.865216    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Jul 09 18:18:50.252136 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:50.252104    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerID="12e271170cc3db897cd0ba3f331bb3ed057f683a8858defaaa497edaee2aa930" exitCode=0
Jul 09 18:18:50.252560 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:50.252181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" event={"ID":"3c6fb84a-eee2-403b-874a-b211c61c21b4","Type":"ContainerDied","Data":"12e271170cc3db897cd0ba3f331bb3ed057f683a8858defaaa497edaee2aa930"}
Jul 09 18:18:50.866675 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:50.866627    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": context deadline exceeded"
Jul 09 18:18:51.082684 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.082650    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:18:51.183818 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.183700    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-kserve-provision-location\") pod \"3c6fb84a-eee2-403b-874a-b211c61c21b4\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") "
Jul 09 18:18:51.183818 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.183750    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5wf6\" (UniqueName: \"kubernetes.io/projected/3c6fb84a-eee2-403b-874a-b211c61c21b4-kube-api-access-q5wf6\") pod \"3c6fb84a-eee2-403b-874a-b211c61c21b4\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") "
Jul 09 18:18:51.183818 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.183771    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-tmp-dir\") pod \"3c6fb84a-eee2-403b-874a-b211c61c21b4\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") "
Jul 09 18:18:51.184113 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.183917    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-model-cache\") pod \"3c6fb84a-eee2-403b-874a-b211c61c21b4\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") "
Jul 09 18:18:51.184113 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.183992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-home\") pod \"3c6fb84a-eee2-403b-874a-b211c61c21b4\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") "
Jul 09 18:18:51.184113 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.184032    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-dshm\") pod \"3c6fb84a-eee2-403b-874a-b211c61c21b4\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") "
Jul 09 18:18:51.184113 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.184063    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6fb84a-eee2-403b-874a-b211c61c21b4-tls-certs\") pod \"3c6fb84a-eee2-403b-874a-b211c61c21b4\" (UID: \"3c6fb84a-eee2-403b-874a-b211c61c21b4\") "
Jul 09 18:18:51.184113 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.184059    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c6fb84a-eee2-403b-874a-b211c61c21b4" (UID: "3c6fb84a-eee2-403b-874a-b211c61c21b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:51.184369 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.184120    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c6fb84a-eee2-403b-874a-b211c61c21b4" (UID: "3c6fb84a-eee2-403b-874a-b211c61c21b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:51.184369 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.184252    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-home" (OuterVolumeSpecName: "home") pod "3c6fb84a-eee2-403b-874a-b211c61c21b4" (UID: "3c6fb84a-eee2-403b-874a-b211c61c21b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:51.192584 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.184384    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:18:51.192584 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.184927    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:18:51.192584 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.184945    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:18:51.192584 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.186921    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6fb84a-eee2-403b-874a-b211c61c21b4-kube-api-access-q5wf6" (OuterVolumeSpecName: "kube-api-access-q5wf6") pod "3c6fb84a-eee2-403b-874a-b211c61c21b4" (UID: "3c6fb84a-eee2-403b-874a-b211c61c21b4"). InnerVolumeSpecName "kube-api-access-q5wf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:51.192584 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.188010    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-dshm" (OuterVolumeSpecName: "dshm") pod "3c6fb84a-eee2-403b-874a-b211c61c21b4" (UID: "3c6fb84a-eee2-403b-874a-b211c61c21b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:51.193363 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.193332    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6fb84a-eee2-403b-874a-b211c61c21b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c6fb84a-eee2-403b-874a-b211c61c21b4" (UID: "3c6fb84a-eee2-403b-874a-b211c61c21b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:51.240589 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.240524    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c6fb84a-eee2-403b-874a-b211c61c21b4" (UID: "3c6fb84a-eee2-403b-874a-b211c61c21b4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:51.259219 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.259185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" event={"ID":"3c6fb84a-eee2-403b-874a-b211c61c21b4","Type":"ContainerDied","Data":"1b79774968b35816a0eeb3175c6dabf38eb3255053783644b82787383fa9a468"}
Jul 09 18:18:51.259636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.259239    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"
Jul 09 18:18:51.259636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.259275    2578 scope.go:117] "RemoveContainer" containerID="12e271170cc3db897cd0ba3f331bb3ed057f683a8858defaaa497edaee2aa930"
Jul 09 18:18:51.279664 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.279641    2578 scope.go:117] "RemoveContainer" containerID="7368a1465935bd0dbaf07164afaecf94537c779c1fe704a58c5538fcebfbf4dd"
Jul 09 18:18:51.283493 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.283466    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"]
Jul 09 18:18:51.286478 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.286342    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:18:51.287274 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.287047    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5wf6\" (UniqueName: \"kubernetes.io/projected/3c6fb84a-eee2-403b-874a-b211c61c21b4-kube-api-access-q5wf6\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:18:51.287274 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.287086    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6fb84a-eee2-403b-874a-b211c61c21b4-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:18:51.287274 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.287102    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6fb84a-eee2-403b-874a-b211c61c21b4-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:18:51.288879 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.288857    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk"]
Jul 09 18:18:51.405717 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.405686    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" path="/var/lib/kubelet/pods/3c6fb84a-eee2-403b-874a-b211c61c21b4/volumes"
Jul 09 18:18:51.867166 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:51.867105    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 09 18:18:52.265432 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:52.265400    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" event={"ID":"26943d6c-548d-47ed-aff8-d9884a7947ac","Type":"ContainerStarted","Data":"11173998cf84f3d8231da7f7302a8dd8de97d8add209de2a7f03c8596968067e"}
Jul 09 18:18:52.295095 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:52.295039    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podStartSLOduration=3.164203306 podStartE2EDuration="53.29501847s" totalImagesPullingTime="46.355701649s" totalInitContainerRuntime="3.775113515s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:52.293102372 +0000 UTC m=+855.480080156" watchObservedRunningTime="2026-07-09 18:18:52.29501847 +0000 UTC m=+855.481996239"
Jul 09 18:18:53.221428 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:53.221393    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:18:57.910108 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:57.910067    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:58.245827 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:18:58.245765    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:00.221748 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:19:00.221709    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:19:00.223167 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:19:00.223139    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 09 18:19:06.572548 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:19:06.572513    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:10.221678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:19:10.221623    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 09 18:19:20.221989 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:19:20.221943    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 09 18:19:30.221847 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:19:30.221707    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 09 18:19:31.656461 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:19:31.656430    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:32.131027 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:19:32.130993    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:37.279001 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:19:37.278966    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:19:37.280582 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:19:37.280555    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:19:37.287753 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:19:37.287725    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:19:37.289670 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:19:37.289645    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:19:40.221958 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:19:40.221909    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 09 18:19:50.222439 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:19:50.222398    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 09 18:20:00.221521 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:20:00.221476    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 09 18:20:01.649683 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:20:01.649648    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:20:10.222349 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:20:10.222300    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 09 18:20:20.222495 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:20:20.222445    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 09 18:20:21.726442 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:20:21.726386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:20:21.948808 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:20:21.948754    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:20:30.222489 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:20:30.222430    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 09 18:20:39.046961 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:20:39.046926    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:20:40.222384 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:20:40.222331    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 09 18:20:50.222059 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:20:50.222008    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 09 18:21:00.235005 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:00.234923    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:21:00.242919 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:00.242884    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:21:04.967915 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:04.967869    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"]
Jul 09 18:21:04.971944 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:04.971913    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"]
Jul 09 18:21:04.981709 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:21:04.981664    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:21:04.981709 ip-10-0-137-79 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:21:04.981709 ip-10-0-137-79 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 09 18:21:04.981709 ip-10-0-137-79 kubenswrapper[2578]:  > containerID="4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175" cmd=["/bin/sleep","15"]
Jul 09 18:21:04.981987 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:04.981753    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67" message="" err=<
Jul 09 18:21:04.981987 ip-10-0-137-79 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:21:04.981987 ip-10-0-137-79 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 09 18:21:04.981987 ip-10-0-137-79 kubenswrapper[2578]:  >
Jul 09 18:21:04.981987 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:21:04.981772    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:21:04.981987 ip-10-0-137-79 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:21:04.981987 ip-10-0-137-79 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 09 18:21:04.981987 ip-10-0-137-79 kubenswrapper[2578]:  > pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67" podUID="89ddf9ed-4010-43ec-9b35-5ad76988e291" containerName="main" containerID="cri-o://4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175"
Jul 09 18:21:04.981987 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:04.981835    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67" podUID="89ddf9ed-4010-43ec-9b35-5ad76988e291" containerName="main" containerID="cri-o://4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175" gracePeriod=60
Jul 09 18:21:04.997783 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:04.997750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:21:05.017176 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:21:05.017151    2578 logging.go:55] [core] [Channel #116 SubChannel #117]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.33:9003", ServerName: "10.132.0.33:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.33:9003: connect: connection refused"
Jul 09 18:21:05.235873 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.235833    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:21:05.370256 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.370221    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ddf9ed-4010-43ec-9b35-5ad76988e291-tls-certs\") pod \"89ddf9ed-4010-43ec-9b35-5ad76988e291\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") "
Jul 09 18:21:05.370456 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.370300    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/89ddf9ed-4010-43ec-9b35-5ad76988e291-tokenizer-uds\") pod \"89ddf9ed-4010-43ec-9b35-5ad76988e291\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") "
Jul 09 18:21:05.370456 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.370336    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6zvr\" (UniqueName: \"kubernetes.io/projected/89ddf9ed-4010-43ec-9b35-5ad76988e291-kube-api-access-r6zvr\") pod \"89ddf9ed-4010-43ec-9b35-5ad76988e291\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") "
Jul 09 18:21:05.370456 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.370376    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89ddf9ed-4010-43ec-9b35-5ad76988e291-kserve-provision-location\") pod \"89ddf9ed-4010-43ec-9b35-5ad76988e291\" (UID: \"89ddf9ed-4010-43ec-9b35-5ad76988e291\") "
Jul 09 18:21:05.370743 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.370633    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ddf9ed-4010-43ec-9b35-5ad76988e291-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "89ddf9ed-4010-43ec-9b35-5ad76988e291" (UID: "89ddf9ed-4010-43ec-9b35-5ad76988e291"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:05.371102 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.371079    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ddf9ed-4010-43ec-9b35-5ad76988e291-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89ddf9ed-4010-43ec-9b35-5ad76988e291" (UID: "89ddf9ed-4010-43ec-9b35-5ad76988e291"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:05.372506 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.372484    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ddf9ed-4010-43ec-9b35-5ad76988e291-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89ddf9ed-4010-43ec-9b35-5ad76988e291" (UID: "89ddf9ed-4010-43ec-9b35-5ad76988e291"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:21:05.372592 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.372526    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ddf9ed-4010-43ec-9b35-5ad76988e291-kube-api-access-r6zvr" (OuterVolumeSpecName: "kube-api-access-r6zvr") pod "89ddf9ed-4010-43ec-9b35-5ad76988e291" (UID: "89ddf9ed-4010-43ec-9b35-5ad76988e291"). InnerVolumeSpecName "kube-api-access-r6zvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:21:05.471653 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.471559    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/89ddf9ed-4010-43ec-9b35-5ad76988e291-tokenizer-uds\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:21:05.471653 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.471594    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6zvr\" (UniqueName: \"kubernetes.io/projected/89ddf9ed-4010-43ec-9b35-5ad76988e291-kube-api-access-r6zvr\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:21:05.471653 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.471604    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89ddf9ed-4010-43ec-9b35-5ad76988e291-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:21:05.471653 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.471614    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ddf9ed-4010-43ec-9b35-5ad76988e291-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:21:05.907480 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.907391    2578 generic.go:356] "Generic (PLEG): container finished" podID="89ddf9ed-4010-43ec-9b35-5ad76988e291" containerID="4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175" exitCode=0
Jul 09 18:21:05.907480 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.907445    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67" event={"ID":"89ddf9ed-4010-43ec-9b35-5ad76988e291","Type":"ContainerDied","Data":"4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175"}
Jul 09 18:21:05.907480 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.907459    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"
Jul 09 18:21:05.907819 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.907512    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67" event={"ID":"89ddf9ed-4010-43ec-9b35-5ad76988e291","Type":"ContainerDied","Data":"40e1e257fc47026b1f89d5b038017b9b07bfb4bc9b392c7582cfeae56ce2acb2"}
Jul 09 18:21:05.907819 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.907532    2578 scope.go:117] "RemoveContainer" containerID="4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175"
Jul 09 18:21:05.916238 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.916219    2578 scope.go:117] "RemoveContainer" containerID="d9bfaf1fe8c269a484a7314ff6cdc932662f27077200e59a5ab86430463a0290"
Jul 09 18:21:05.924320 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.924299    2578 scope.go:117] "RemoveContainer" containerID="4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175"
Jul 09 18:21:05.924638 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:21:05.924621    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175\": container with ID starting with 4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175 not found: ID does not exist" containerID="4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175"
Jul 09 18:21:05.924695 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.924647    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175"} err="failed to get container status \"4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175\": rpc error: code = NotFound desc = could not find container \"4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175\": container with ID starting with 4fdaba10c5f331479d5df590e4f9c4be6074e2b4165aa831e9a4f919ae16e175 not found: ID does not exist"
Jul 09 18:21:05.924695 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.924665    2578 scope.go:117] "RemoveContainer" containerID="d9bfaf1fe8c269a484a7314ff6cdc932662f27077200e59a5ab86430463a0290"
Jul 09 18:21:05.924945 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:21:05.924926    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9bfaf1fe8c269a484a7314ff6cdc932662f27077200e59a5ab86430463a0290\": container with ID starting with d9bfaf1fe8c269a484a7314ff6cdc932662f27077200e59a5ab86430463a0290 not found: ID does not exist" containerID="d9bfaf1fe8c269a484a7314ff6cdc932662f27077200e59a5ab86430463a0290"
Jul 09 18:21:05.925010 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.924949    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9bfaf1fe8c269a484a7314ff6cdc932662f27077200e59a5ab86430463a0290"} err="failed to get container status \"d9bfaf1fe8c269a484a7314ff6cdc932662f27077200e59a5ab86430463a0290\": rpc error: code = NotFound desc = could not find container \"d9bfaf1fe8c269a484a7314ff6cdc932662f27077200e59a5ab86430463a0290\": container with ID starting with d9bfaf1fe8c269a484a7314ff6cdc932662f27077200e59a5ab86430463a0290 not found: ID does not exist"
Jul 09 18:21:05.928332 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.928305    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"]
Jul 09 18:21:05.931007 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:05.930984    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67"]
Jul 09 18:21:06.016930 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:06.016878    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67" podUID="89ddf9ed-4010-43ec-9b35-5ad76988e291" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.33:9003\" within 1s: context deadline exceeded"
Jul 09 18:21:06.902065 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:06.902032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:21:07.403935 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:07.403903    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89ddf9ed-4010-43ec-9b35-5ad76988e291" path="/var/lib/kubelet/pods/89ddf9ed-4010-43ec-9b35-5ad76988e291/volumes"
Jul 09 18:21:09.110675 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:09.110641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:21:19.993911 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:19.993868    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main" containerID="cri-o://11173998cf84f3d8231da7f7302a8dd8de97d8add209de2a7f03c8596968067e" gracePeriod=45
Jul 09 18:21:20.236230 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.236182    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 09 18:21:20.742921 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.742895    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:21:20.907878 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.907742    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-kserve-provision-location\") pod \"26943d6c-548d-47ed-aff8-d9884a7947ac\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") "
Jul 09 18:21:20.907878 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.907848    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-model-cache\") pod \"26943d6c-548d-47ed-aff8-d9884a7947ac\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") "
Jul 09 18:21:20.907878 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.907878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-tmp-dir\") pod \"26943d6c-548d-47ed-aff8-d9884a7947ac\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") "
Jul 09 18:21:20.908153 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.907905    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-dshm\") pod \"26943d6c-548d-47ed-aff8-d9884a7947ac\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") "
Jul 09 18:21:20.908153 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.907941    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8jgw\" (UniqueName: \"kubernetes.io/projected/26943d6c-548d-47ed-aff8-d9884a7947ac-kube-api-access-t8jgw\") pod \"26943d6c-548d-47ed-aff8-d9884a7947ac\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") "
Jul 09 18:21:20.908153 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.907965    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26943d6c-548d-47ed-aff8-d9884a7947ac-tls-certs\") pod \"26943d6c-548d-47ed-aff8-d9884a7947ac\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") "
Jul 09 18:21:20.908153 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.907985    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-home\") pod \"26943d6c-548d-47ed-aff8-d9884a7947ac\" (UID: \"26943d6c-548d-47ed-aff8-d9884a7947ac\") "
Jul 09 18:21:20.908153 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.908107    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "26943d6c-548d-47ed-aff8-d9884a7947ac" (UID: "26943d6c-548d-47ed-aff8-d9884a7947ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:20.908371 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.908294    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:21:20.908898 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.908869    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-home" (OuterVolumeSpecName: "home") pod "26943d6c-548d-47ed-aff8-d9884a7947ac" (UID: "26943d6c-548d-47ed-aff8-d9884a7947ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:20.910098 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.910068    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-dshm" (OuterVolumeSpecName: "dshm") pod "26943d6c-548d-47ed-aff8-d9884a7947ac" (UID: "26943d6c-548d-47ed-aff8-d9884a7947ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:20.910557 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.910536    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26943d6c-548d-47ed-aff8-d9884a7947ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26943d6c-548d-47ed-aff8-d9884a7947ac" (UID: "26943d6c-548d-47ed-aff8-d9884a7947ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:21:20.910611 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.910558    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26943d6c-548d-47ed-aff8-d9884a7947ac-kube-api-access-t8jgw" (OuterVolumeSpecName: "kube-api-access-t8jgw") pod "26943d6c-548d-47ed-aff8-d9884a7947ac" (UID: "26943d6c-548d-47ed-aff8-d9884a7947ac"). InnerVolumeSpecName "kube-api-access-t8jgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:21:20.927264 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.927228    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26943d6c-548d-47ed-aff8-d9884a7947ac" (UID: "26943d6c-548d-47ed-aff8-d9884a7947ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:20.966932 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.966885    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26943d6c-548d-47ed-aff8-d9884a7947ac" (UID: "26943d6c-548d-47ed-aff8-d9884a7947ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:20.975819 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.975763    2578 generic.go:356] "Generic (PLEG): container finished" podID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerID="11173998cf84f3d8231da7f7302a8dd8de97d8add209de2a7f03c8596968067e" exitCode=0
Jul 09 18:21:20.975968 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.975825    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"
Jul 09 18:21:20.975968 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.975829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" event={"ID":"26943d6c-548d-47ed-aff8-d9884a7947ac","Type":"ContainerDied","Data":"11173998cf84f3d8231da7f7302a8dd8de97d8add209de2a7f03c8596968067e"}
Jul 09 18:21:20.975968 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.975892    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq" event={"ID":"26943d6c-548d-47ed-aff8-d9884a7947ac","Type":"ContainerDied","Data":"b207aabf3f9130a6731cdbbff721a7184a60564720fb043118466e984d153edd"}
Jul 09 18:21:20.975968 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.975916    2578 scope.go:117] "RemoveContainer" containerID="11173998cf84f3d8231da7f7302a8dd8de97d8add209de2a7f03c8596968067e"
Jul 09 18:21:20.984664 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.984645    2578 scope.go:117] "RemoveContainer" containerID="a503afafc82c79b8f22dab313ef6d072d67edf5ad79c736349ebaec7fa7dc1af"
Jul 09 18:21:20.993128 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.993100    2578 scope.go:117] "RemoveContainer" containerID="11173998cf84f3d8231da7f7302a8dd8de97d8add209de2a7f03c8596968067e"
Jul 09 18:21:20.993402 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:21:20.993382    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11173998cf84f3d8231da7f7302a8dd8de97d8add209de2a7f03c8596968067e\": container with ID starting with 11173998cf84f3d8231da7f7302a8dd8de97d8add209de2a7f03c8596968067e not found: ID does not exist" containerID="11173998cf84f3d8231da7f7302a8dd8de97d8add209de2a7f03c8596968067e"
Jul 09 18:21:20.993457 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.993412    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11173998cf84f3d8231da7f7302a8dd8de97d8add209de2a7f03c8596968067e"} err="failed to get container status \"11173998cf84f3d8231da7f7302a8dd8de97d8add209de2a7f03c8596968067e\": rpc error: code = NotFound desc = could not find container \"11173998cf84f3d8231da7f7302a8dd8de97d8add209de2a7f03c8596968067e\": container with ID starting with 11173998cf84f3d8231da7f7302a8dd8de97d8add209de2a7f03c8596968067e not found: ID does not exist"
Jul 09 18:21:20.993457 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.993429    2578 scope.go:117] "RemoveContainer" containerID="a503afafc82c79b8f22dab313ef6d072d67edf5ad79c736349ebaec7fa7dc1af"
Jul 09 18:21:20.993664 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:21:20.993645    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a503afafc82c79b8f22dab313ef6d072d67edf5ad79c736349ebaec7fa7dc1af\": container with ID starting with a503afafc82c79b8f22dab313ef6d072d67edf5ad79c736349ebaec7fa7dc1af not found: ID does not exist" containerID="a503afafc82c79b8f22dab313ef6d072d67edf5ad79c736349ebaec7fa7dc1af"
Jul 09 18:21:20.993719 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.993674    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a503afafc82c79b8f22dab313ef6d072d67edf5ad79c736349ebaec7fa7dc1af"} err="failed to get container status \"a503afafc82c79b8f22dab313ef6d072d67edf5ad79c736349ebaec7fa7dc1af\": rpc error: code = NotFound desc = could not find container \"a503afafc82c79b8f22dab313ef6d072d67edf5ad79c736349ebaec7fa7dc1af\": container with ID starting with a503afafc82c79b8f22dab313ef6d072d67edf5ad79c736349ebaec7fa7dc1af not found: ID does not exist"
Jul 09 18:21:20.997972 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:20.997948    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"]
Jul 09 18:21:21.002514 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:21.002489    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq"]
Jul 09 18:21:21.009278 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:21.009257    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:21:21.009278 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:21.009280    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:21:21.009386 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:21.009289    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8jgw\" (UniqueName: \"kubernetes.io/projected/26943d6c-548d-47ed-aff8-d9884a7947ac-kube-api-access-t8jgw\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:21:21.009386 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:21.009299    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26943d6c-548d-47ed-aff8-d9884a7947ac-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:21:21.009386 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:21.009308    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:21:21.009386 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:21.009316    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26943d6c-548d-47ed-aff8-d9884a7947ac-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:21:21.402579 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:21.402544    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" path="/var/lib/kubelet/pods/26943d6c-548d-47ed-aff8-d9884a7947ac/volumes"
Jul 09 18:21:27.385786 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.385752    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"]
Jul 09 18:21:27.386194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386069    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main"
Jul 09 18:21:27.386194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386082    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main"
Jul 09 18:21:27.386194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386099    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ddf9ed-4010-43ec-9b35-5ad76988e291" containerName="storage-initializer"
Jul 09 18:21:27.386194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386104    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ddf9ed-4010-43ec-9b35-5ad76988e291" containerName="storage-initializer"
Jul 09 18:21:27.386194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386114    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ddf9ed-4010-43ec-9b35-5ad76988e291" containerName="main"
Jul 09 18:21:27.386194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386120    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ddf9ed-4010-43ec-9b35-5ad76988e291" containerName="main"
Jul 09 18:21:27.386194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386127    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="storage-initializer"
Jul 09 18:21:27.386194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386133    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="storage-initializer"
Jul 09 18:21:27.386194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386140    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="storage-initializer"
Jul 09 18:21:27.386194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386145    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="storage-initializer"
Jul 09 18:21:27.386194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386154    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main"
Jul 09 18:21:27.386194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386158    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main"
Jul 09 18:21:27.386549 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386206    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="89ddf9ed-4010-43ec-9b35-5ad76988e291" containerName="main"
Jul 09 18:21:27.386549 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386214    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="26943d6c-548d-47ed-aff8-d9884a7947ac" containerName="main"
Jul 09 18:21:27.386549 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.386222    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c6fb84a-eee2-403b-874a-b211c61c21b4" containerName="main"
Jul 09 18:21:27.389152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.389134    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.392911 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.392888    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 09 18:21:27.393036 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.392963    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-s8z92\""
Jul 09 18:21:27.412950 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.412915    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"]
Jul 09 18:21:27.435395 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.435360    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:21:27.565931 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.565889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.565931 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.565939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.566149 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.565971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.566149 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.565992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b937b9-6ead-4d26-ac10-f5439a52855a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.566149 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.566029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.566149 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.566101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvpk\" (UniqueName: \"kubernetes.io/projected/43b937b9-6ead-4d26-ac10-f5439a52855a-kube-api-access-mxvpk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.667465 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.667372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.667465 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.667424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.667465 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.667448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.667465 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.667467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b937b9-6ead-4d26-ac10-f5439a52855a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.667835 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.667509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.667835 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.667549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvpk\" (UniqueName: \"kubernetes.io/projected/43b937b9-6ead-4d26-ac10-f5439a52855a-kube-api-access-mxvpk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.667956 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.667926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.668011 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.667936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.668070 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.668036    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.669663 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.669638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.670051 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.670031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b937b9-6ead-4d26-ac10-f5439a52855a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.677252 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.677227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvpk\" (UniqueName: \"kubernetes.io/projected/43b937b9-6ead-4d26-ac10-f5439a52855a-kube-api-access-mxvpk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.699551 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.699519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:27.736571 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.736536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:21:27.835203 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:27.835146    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"]
Jul 09 18:21:27.838376 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:21:27.838335    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b937b9_6ead_4d26_ac10_f5439a52855a.slice/crio-233ad79efdba798cb1ca507cd41650f728eaf2509078fb7d03f659bbfbe4639c WatchSource:0}: Error finding container 233ad79efdba798cb1ca507cd41650f728eaf2509078fb7d03f659bbfbe4639c: Status 404 returned error can't find the container with id 233ad79efdba798cb1ca507cd41650f728eaf2509078fb7d03f659bbfbe4639c
Jul 09 18:21:28.008646 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:28.008604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk" event={"ID":"43b937b9-6ead-4d26-ac10-f5439a52855a","Type":"ContainerStarted","Data":"3e586879bfab9bb656f4ab436d1c52d3a9b7d86bc5d8ffa91ebf4b1689113b4b"}
Jul 09 18:21:28.008831 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:28.008663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk" event={"ID":"43b937b9-6ead-4d26-ac10-f5439a52855a","Type":"ContainerStarted","Data":"233ad79efdba798cb1ca507cd41650f728eaf2509078fb7d03f659bbfbe4639c"}
Jul 09 18:21:28.029042 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:28.028991    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk" podStartSLOduration=1.028974781 podStartE2EDuration="1.028974781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:21:28.027917106 +0000 UTC m=+1011.214894896" watchObservedRunningTime="2026-07-09 18:21:28.028974781 +0000 UTC m=+1011.215952550"
Jul 09 18:21:28.700600 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:28.700557    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:28.997068 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:28.996974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:21:29.098471 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:29.098436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:21:37.700004 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:37.699961    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:37.712698 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:37.712670    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:38.062872 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:38.062771    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:21:48.258202 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.258166    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"]
Jul 09 18:21:48.261581 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.261557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.264916 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.264881    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:21:48.265054 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.264894    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 09 18:21:48.274010 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.273981    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"]
Jul 09 18:21:48.292706 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.292673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:21:48.327040 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.327003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-dshm\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.327234 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.327063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.327234 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.327104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-model-cache\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.327234 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.327122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-tmp-dir\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.327234 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.327209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5psx\" (UniqueName: \"kubernetes.io/projected/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-kube-api-access-z5psx\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.327370 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.327250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-home\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.327370 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.327280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-tls-certs\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.428726 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.428688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.428952 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.428744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-model-cache\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.428952 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.428763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-tmp-dir\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.428952 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.428859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5psx\" (UniqueName: \"kubernetes.io/projected/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-kube-api-access-z5psx\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.428952 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.428920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-home\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.429193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.428971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-tls-certs\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.429193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.429007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-dshm\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.429193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.429181    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.429360 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.429228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-model-cache\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.429360 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.429264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-tmp-dir\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.429466 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.429378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-home\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.431347 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.431322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-dshm\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.431516 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.431501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-tls-certs\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.439102 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.439079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5psx\" (UniqueName: \"kubernetes.io/projected/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-kube-api-access-z5psx\") pod \"auth-invalid-token-test-kserve-7d5c575bbb-7mwr2\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.553767 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.553679    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:21:48.571725 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.571685    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:48.716963 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:48.716930    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"]
Jul 09 18:21:48.720060 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:21:48.720018    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afd0e3d_199a_4bf1_b69a_7e97d480ab3c.slice/crio-ec60a08d51c1ce093f6277ecbd7c69ac003e1397da5e63b834e8b35e8ec2070d WatchSource:0}: Error finding container ec60a08d51c1ce093f6277ecbd7c69ac003e1397da5e63b834e8b35e8ec2070d: Status 404 returned error can't find the container with id ec60a08d51c1ce093f6277ecbd7c69ac003e1397da5e63b834e8b35e8ec2070d
Jul 09 18:21:49.102047 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:49.102010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" event={"ID":"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c","Type":"ContainerStarted","Data":"10f4e16936b17095baed513c13464eae721d184c301a7bb563234595b03d362c"}
Jul 09 18:21:49.102249 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:49.102067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" event={"ID":"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c","Type":"ContainerStarted","Data":"ec60a08d51c1ce093f6277ecbd7c69ac003e1397da5e63b834e8b35e8ec2070d"}
Jul 09 18:21:52.117595 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:52.117548    2578 generic.go:356] "Generic (PLEG): container finished" podID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerID="10f4e16936b17095baed513c13464eae721d184c301a7bb563234595b03d362c" exitCode=0
Jul 09 18:21:52.118002 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:52.117622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" event={"ID":"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c","Type":"ContainerDied","Data":"10f4e16936b17095baed513c13464eae721d184c301a7bb563234595b03d362c"}
Jul 09 18:21:53.124181 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:53.124146    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" event={"ID":"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c","Type":"ContainerStarted","Data":"3f0269b5163d307f1dea553ef81aae8cefdb5c74335d2be0a580ac4162108cb0"}
Jul 09 18:21:53.147734 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:53.147670    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" podStartSLOduration=1.8883009149999999 podStartE2EDuration="5.147651788s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.259350873s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:21:53.144983166 +0000 UTC m=+1036.331960953" watchObservedRunningTime="2026-07-09 18:21:53.147651788 +0000 UTC m=+1036.334629628"
Jul 09 18:21:53.572869 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:53.572828    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:58.572582 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:58.572548    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:21:58.574307 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:21:58.574281    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 09 18:22:00.490848 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:00.490814    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:22:08.573149 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:08.573100    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 09 18:22:13.061536 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:13.061499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:22:17.027884 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:17.027853    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"]
Jul 09 18:22:17.065327 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:17.065290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:22:17.156756 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:17.156713    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:22:18.572952 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:18.572896    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 09 18:22:28.572484 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:28.572388    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 09 18:22:32.048134 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.048069    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk" podUID="43b937b9-6ead-4d26-ac10-f5439a52855a" containerName="main" containerID="cri-o://3e586879bfab9bb656f4ab436d1c52d3a9b7d86bc5d8ffa91ebf4b1689113b4b" gracePeriod=45
Jul 09 18:22:32.051921 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.051887    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk" podUID="43b937b9-6ead-4d26-ac10-f5439a52855a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Jul 09 18:22:32.302334 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.302260    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:22:32.314045 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.314005    2578 generic.go:356] "Generic (PLEG): container finished" podID="43b937b9-6ead-4d26-ac10-f5439a52855a" containerID="3e586879bfab9bb656f4ab436d1c52d3a9b7d86bc5d8ffa91ebf4b1689113b4b" exitCode=0
Jul 09 18:22:32.314227 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.314062    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"
Jul 09 18:22:32.314227 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.314085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk" event={"ID":"43b937b9-6ead-4d26-ac10-f5439a52855a","Type":"ContainerDied","Data":"3e586879bfab9bb656f4ab436d1c52d3a9b7d86bc5d8ffa91ebf4b1689113b4b"}
Jul 09 18:22:32.314227 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.314150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk" event={"ID":"43b937b9-6ead-4d26-ac10-f5439a52855a","Type":"ContainerDied","Data":"233ad79efdba798cb1ca507cd41650f728eaf2509078fb7d03f659bbfbe4639c"}
Jul 09 18:22:32.314227 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.314180    2578 scope.go:117] "RemoveContainer" containerID="3e586879bfab9bb656f4ab436d1c52d3a9b7d86bc5d8ffa91ebf4b1689113b4b"
Jul 09 18:22:32.324969 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.324935    2578 scope.go:117] "RemoveContainer" containerID="3e586879bfab9bb656f4ab436d1c52d3a9b7d86bc5d8ffa91ebf4b1689113b4b"
Jul 09 18:22:32.325353 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:22:32.325324    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e586879bfab9bb656f4ab436d1c52d3a9b7d86bc5d8ffa91ebf4b1689113b4b\": container with ID starting with 3e586879bfab9bb656f4ab436d1c52d3a9b7d86bc5d8ffa91ebf4b1689113b4b not found: ID does not exist" containerID="3e586879bfab9bb656f4ab436d1c52d3a9b7d86bc5d8ffa91ebf4b1689113b4b"
Jul 09 18:22:32.325492 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.325367    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e586879bfab9bb656f4ab436d1c52d3a9b7d86bc5d8ffa91ebf4b1689113b4b"} err="failed to get container status \"3e586879bfab9bb656f4ab436d1c52d3a9b7d86bc5d8ffa91ebf4b1689113b4b\": rpc error: code = NotFound desc = could not find container \"3e586879bfab9bb656f4ab436d1c52d3a9b7d86bc5d8ffa91ebf4b1689113b4b\": container with ID starting with 3e586879bfab9bb656f4ab436d1c52d3a9b7d86bc5d8ffa91ebf4b1689113b4b not found: ID does not exist"
Jul 09 18:22:32.343046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.343012    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-tmp-dir\") pod \"43b937b9-6ead-4d26-ac10-f5439a52855a\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") "
Jul 09 18:22:32.343239 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.343061    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-home\") pod \"43b937b9-6ead-4d26-ac10-f5439a52855a\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") "
Jul 09 18:22:32.343239 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.343083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-model-cache\") pod \"43b937b9-6ead-4d26-ac10-f5439a52855a\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") "
Jul 09 18:22:32.343239 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.343130    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxvpk\" (UniqueName: \"kubernetes.io/projected/43b937b9-6ead-4d26-ac10-f5439a52855a-kube-api-access-mxvpk\") pod \"43b937b9-6ead-4d26-ac10-f5439a52855a\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") "
Jul 09 18:22:32.343239 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.343164    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-dshm\") pod \"43b937b9-6ead-4d26-ac10-f5439a52855a\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") "
Jul 09 18:22:32.343239 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.343198    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b937b9-6ead-4d26-ac10-f5439a52855a-tls-certs\") pod \"43b937b9-6ead-4d26-ac10-f5439a52855a\" (UID: \"43b937b9-6ead-4d26-ac10-f5439a52855a\") "
Jul 09 18:22:32.343565 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.343337    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43b937b9-6ead-4d26-ac10-f5439a52855a" (UID: "43b937b9-6ead-4d26-ac10-f5439a52855a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:32.343565 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.343431    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-model-cache" (OuterVolumeSpecName: "model-cache") pod "43b937b9-6ead-4d26-ac10-f5439a52855a" (UID: "43b937b9-6ead-4d26-ac10-f5439a52855a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:32.343565 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.343480    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:22:32.343565 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.343547    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-home" (OuterVolumeSpecName: "home") pod "43b937b9-6ead-4d26-ac10-f5439a52855a" (UID: "43b937b9-6ead-4d26-ac10-f5439a52855a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:32.345491 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.345463    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b937b9-6ead-4d26-ac10-f5439a52855a-kube-api-access-mxvpk" (OuterVolumeSpecName: "kube-api-access-mxvpk") pod "43b937b9-6ead-4d26-ac10-f5439a52855a" (UID: "43b937b9-6ead-4d26-ac10-f5439a52855a"). InnerVolumeSpecName "kube-api-access-mxvpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:22:32.345763 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.345745    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-dshm" (OuterVolumeSpecName: "dshm") pod "43b937b9-6ead-4d26-ac10-f5439a52855a" (UID: "43b937b9-6ead-4d26-ac10-f5439a52855a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:32.345889 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.345769    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b937b9-6ead-4d26-ac10-f5439a52855a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43b937b9-6ead-4d26-ac10-f5439a52855a" (UID: "43b937b9-6ead-4d26-ac10-f5439a52855a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:22:32.444818 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.444770    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:22:32.445000 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.444848    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:22:32.445000 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.444881    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxvpk\" (UniqueName: \"kubernetes.io/projected/43b937b9-6ead-4d26-ac10-f5439a52855a-kube-api-access-mxvpk\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:22:32.445000 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.444900    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b937b9-6ead-4d26-ac10-f5439a52855a-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:22:32.445000 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.444914    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b937b9-6ead-4d26-ac10-f5439a52855a-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:22:32.637820 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.637773    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"]
Jul 09 18:22:32.641411 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:32.641381    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk"]
Jul 09 18:22:33.402835 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:33.402785    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43b937b9-6ead-4d26-ac10-f5439a52855a" path="/var/lib/kubelet/pods/43b937b9-6ead-4d26-ac10-f5439a52855a/volumes"
Jul 09 18:22:36.890065 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:36.890028    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:22:36.950981 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:36.950944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:22:38.572588 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:38.572548    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 09 18:22:40.805878 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:40.805827    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:22:48.572764 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:48.572720    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 09 18:22:58.572807 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:22:58.572760    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 09 18:23:07.611861 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:23:07.611785    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:23:07.705933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:23:07.705862    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:23:08.572269 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:23:08.572220    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 09 18:23:18.573142 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:23:18.573098    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 09 18:23:19.767597 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:23:19.767558    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:23:28.572209 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:23:28.572156    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 09 18:23:38.572492 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:23:38.572450    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 09 18:23:48.573014 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:23:48.572965    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 09 18:23:52.397909 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:23:52.397876    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:23:58.582983 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:23:58.582899    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:23:58.591466 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:23:58.591441    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:23:59.724742 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:23:59.724707    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"]
Jul 09 18:23:59.731099 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:23:59.731069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:23:59.794188 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:23:59.794155    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:14.759542 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:14.759477    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main" containerID="cri-o://3f0269b5163d307f1dea553ef81aae8cefdb5c74335d2be0a580ac4162108cb0" gracePeriod=45
Jul 09 18:24:15.509022 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.508994    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:24:15.575168 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.575081    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-dshm\") pod \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") "
Jul 09 18:24:15.575168 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.575133    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-tls-certs\") pod \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") "
Jul 09 18:24:15.575168 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.575155    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-home\") pod \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") "
Jul 09 18:24:15.575430 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.575189    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-tmp-dir\") pod \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") "
Jul 09 18:24:15.575430 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.575249    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-kserve-provision-location\") pod \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") "
Jul 09 18:24:15.575430 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.575288    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-model-cache\") pod \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") "
Jul 09 18:24:15.575430 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.575319    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5psx\" (UniqueName: \"kubernetes.io/projected/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-kube-api-access-z5psx\") pod \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\" (UID: \"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c\") "
Jul 09 18:24:15.575646 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.575592    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-model-cache" (OuterVolumeSpecName: "model-cache") pod "9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" (UID: "9afd0e3d-199a-4bf1-b69a-7e97d480ab3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:15.575966 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.575937    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-home" (OuterVolumeSpecName: "home") pod "9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" (UID: "9afd0e3d-199a-4bf1-b69a-7e97d480ab3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:15.577396 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.577372    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" (UID: "9afd0e3d-199a-4bf1-b69a-7e97d480ab3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:15.577509 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.577486    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-kube-api-access-z5psx" (OuterVolumeSpecName: "kube-api-access-z5psx") pod "9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" (UID: "9afd0e3d-199a-4bf1-b69a-7e97d480ab3c"). InnerVolumeSpecName "kube-api-access-z5psx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:15.577749 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.577736    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-dshm" (OuterVolumeSpecName: "dshm") pod "9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" (UID: "9afd0e3d-199a-4bf1-b69a-7e97d480ab3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:15.593446 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.593407    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" (UID: "9afd0e3d-199a-4bf1-b69a-7e97d480ab3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:15.638192 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.638147    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" (UID: "9afd0e3d-199a-4bf1-b69a-7e97d480ab3c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:15.676914 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.676872    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:24:15.676914 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.676906    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:24:15.676914 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.676916    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:24:15.677124 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.676924    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:24:15.677124 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.676933    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:24:15.677124 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.676942    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:24:15.677124 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.676950    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5psx\" (UniqueName: \"kubernetes.io/projected/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c-kube-api-access-z5psx\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:24:15.808189 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.808144    2578 generic.go:356] "Generic (PLEG): container finished" podID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerID="3f0269b5163d307f1dea553ef81aae8cefdb5c74335d2be0a580ac4162108cb0" exitCode=0
Jul 09 18:24:15.808646 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.808315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" event={"ID":"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c","Type":"ContainerDied","Data":"3f0269b5163d307f1dea553ef81aae8cefdb5c74335d2be0a580ac4162108cb0"}
Jul 09 18:24:15.808646 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.808375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2" event={"ID":"9afd0e3d-199a-4bf1-b69a-7e97d480ab3c","Type":"ContainerDied","Data":"ec60a08d51c1ce093f6277ecbd7c69ac003e1397da5e63b834e8b35e8ec2070d"}
Jul 09 18:24:15.808646 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.808406    2578 scope.go:117] "RemoveContainer" containerID="3f0269b5163d307f1dea553ef81aae8cefdb5c74335d2be0a580ac4162108cb0"
Jul 09 18:24:15.808646 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.808589    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"
Jul 09 18:24:15.820135 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.820110    2578 scope.go:117] "RemoveContainer" containerID="10f4e16936b17095baed513c13464eae721d184c301a7bb563234595b03d362c"
Jul 09 18:24:15.829190 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.829169    2578 scope.go:117] "RemoveContainer" containerID="3f0269b5163d307f1dea553ef81aae8cefdb5c74335d2be0a580ac4162108cb0"
Jul 09 18:24:15.829528 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:24:15.829508    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0269b5163d307f1dea553ef81aae8cefdb5c74335d2be0a580ac4162108cb0\": container with ID starting with 3f0269b5163d307f1dea553ef81aae8cefdb5c74335d2be0a580ac4162108cb0 not found: ID does not exist" containerID="3f0269b5163d307f1dea553ef81aae8cefdb5c74335d2be0a580ac4162108cb0"
Jul 09 18:24:15.829566 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.829539    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0269b5163d307f1dea553ef81aae8cefdb5c74335d2be0a580ac4162108cb0"} err="failed to get container status \"3f0269b5163d307f1dea553ef81aae8cefdb5c74335d2be0a580ac4162108cb0\": rpc error: code = NotFound desc = could not find container \"3f0269b5163d307f1dea553ef81aae8cefdb5c74335d2be0a580ac4162108cb0\": container with ID starting with 3f0269b5163d307f1dea553ef81aae8cefdb5c74335d2be0a580ac4162108cb0 not found: ID does not exist"
Jul 09 18:24:15.829566 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.829559    2578 scope.go:117] "RemoveContainer" containerID="10f4e16936b17095baed513c13464eae721d184c301a7bb563234595b03d362c"
Jul 09 18:24:15.829838 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:24:15.829816    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10f4e16936b17095baed513c13464eae721d184c301a7bb563234595b03d362c\": container with ID starting with 10f4e16936b17095baed513c13464eae721d184c301a7bb563234595b03d362c not found: ID does not exist" containerID="10f4e16936b17095baed513c13464eae721d184c301a7bb563234595b03d362c"
Jul 09 18:24:15.829882 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.829845    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10f4e16936b17095baed513c13464eae721d184c301a7bb563234595b03d362c"} err="failed to get container status \"10f4e16936b17095baed513c13464eae721d184c301a7bb563234595b03d362c\": rpc error: code = NotFound desc = could not find container \"10f4e16936b17095baed513c13464eae721d184c301a7bb563234595b03d362c\": container with ID starting with 10f4e16936b17095baed513c13464eae721d184c301a7bb563234595b03d362c not found: ID does not exist"
Jul 09 18:24:15.837782 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.837748    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"]
Jul 09 18:24:15.848151 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:15.848119    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2"]
Jul 09 18:24:17.402908 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:17.402874    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" path="/var/lib/kubelet/pods/9afd0e3d-199a-4bf1-b69a-7e97d480ab3c/volumes"
Jul 09 18:24:22.667760 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.667725    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"]
Jul 09 18:24:22.668223 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.668039    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b937b9-6ead-4d26-ac10-f5439a52855a" containerName="main"
Jul 09 18:24:22.668223 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.668050    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b937b9-6ead-4d26-ac10-f5439a52855a" containerName="main"
Jul 09 18:24:22.668223 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.668069    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="storage-initializer"
Jul 09 18:24:22.668223 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.668075    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="storage-initializer"
Jul 09 18:24:22.668223 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.668082    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main"
Jul 09 18:24:22.668223 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.668088    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main"
Jul 09 18:24:22.668223 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.668134    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="43b937b9-6ead-4d26-ac10-f5439a52855a" containerName="main"
Jul 09 18:24:22.668223 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.668143    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9afd0e3d-199a-4bf1-b69a-7e97d480ab3c" containerName="main"
Jul 09 18:24:22.671258 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.671237    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.673983 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.673960    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 09 18:24:22.674178 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.673961    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:24:22.674178 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.673961    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-s8z92\""
Jul 09 18:24:22.683265 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.683231    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"]
Jul 09 18:24:22.705490 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.705448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:22.841896 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.841864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-model-cache\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.841896 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.841897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996d2ab5-ede1-4c69-866c-221319c86ccf-tls-certs\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.842121 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.841927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-dshm\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.842121 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.841981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkqf8\" (UniqueName: \"kubernetes.io/projected/996d2ab5-ede1-4c69-866c-221319c86ccf-kube-api-access-jkqf8\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.842121 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.842055    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-kserve-provision-location\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.842121 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.842080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-home\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.842121 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.842111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-tmp-dir\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.943569 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.943469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-model-cache\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.943569 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.943520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996d2ab5-ede1-4c69-866c-221319c86ccf-tls-certs\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.943729 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.943642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-dshm\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.943729 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.943693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkqf8\" (UniqueName: \"kubernetes.io/projected/996d2ab5-ede1-4c69-866c-221319c86ccf-kube-api-access-jkqf8\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.943897 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.943768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-kserve-provision-location\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.943897 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.943826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-home\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.943897 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.943876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-tmp-dir\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.944066 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.944005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-model-cache\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.944171 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.944146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-kserve-provision-location\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.944236 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.944213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-home\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.944236 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.944225    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-tmp-dir\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.945854 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.945836    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-dshm\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.946145 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.946128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996d2ab5-ede1-4c69-866c-221319c86ccf-tls-certs\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.952907 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.952881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkqf8\" (UniqueName: \"kubernetes.io/projected/996d2ab5-ede1-4c69-866c-221319c86ccf-kube-api-access-jkqf8\") pod \"auth-disabled-test-kserve-6bc7867d76-llhg4\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:22.983135 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:22.983097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:23.027496 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:23.027466    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:23.128311 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:23.128286    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"]
Jul 09 18:24:23.130221 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:24:23.130189    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996d2ab5_ede1_4c69_866c_221319c86ccf.slice/crio-a2e89b9516864218cfe2b65f9f59555b32ca06946ff165dd71e05d1a7fa68a2b WatchSource:0}: Error finding container a2e89b9516864218cfe2b65f9f59555b32ca06946ff165dd71e05d1a7fa68a2b: Status 404 returned error can't find the container with id a2e89b9516864218cfe2b65f9f59555b32ca06946ff165dd71e05d1a7fa68a2b
Jul 09 18:24:23.845168 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:23.845129    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" event={"ID":"996d2ab5-ede1-4c69-866c-221319c86ccf","Type":"ContainerStarted","Data":"b9b26adcc18e8efa0110fea99708a7d6bb9ca11c11e498ca6aa9d2d99b15ffc9"}
Jul 09 18:24:23.845670 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:23.845193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" event={"ID":"996d2ab5-ede1-4c69-866c-221319c86ccf","Type":"ContainerStarted","Data":"a2e89b9516864218cfe2b65f9f59555b32ca06946ff165dd71e05d1a7fa68a2b"}
Jul 09 18:24:26.859415 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:26.859379    2578 generic.go:356] "Generic (PLEG): container finished" podID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerID="b9b26adcc18e8efa0110fea99708a7d6bb9ca11c11e498ca6aa9d2d99b15ffc9" exitCode=0
Jul 09 18:24:26.859911 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:26.859424    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" event={"ID":"996d2ab5-ede1-4c69-866c-221319c86ccf","Type":"ContainerDied","Data":"b9b26adcc18e8efa0110fea99708a7d6bb9ca11c11e498ca6aa9d2d99b15ffc9"}
Jul 09 18:24:27.866532 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:27.866494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" event={"ID":"996d2ab5-ede1-4c69-866c-221319c86ccf","Type":"ContainerStarted","Data":"d6b975ff3d48c37726d6f301cb561ca6ea3f2a76a8383b99a16cb8fe086c484c"}
Jul 09 18:24:27.896064 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:27.895997    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podStartSLOduration=2.42698007 podStartE2EDuration="5.895976237s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.468996167s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:24:27.894566978 +0000 UTC m=+1191.081544747" watchObservedRunningTime="2026-07-09 18:24:27.895976237 +0000 UTC m=+1191.082954058"
Jul 09 18:24:27.983736 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:27.983691    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:32.983429 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:32.983390    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:24:32.985083 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:32.985050    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 09 18:24:37.308873 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:37.308841    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:24:37.312119 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:37.312095    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:24:37.317992 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:37.317952    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:24:37.321010 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:37.320991    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:24:38.193085 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:38.193049    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:42.984306 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:42.984262    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 09 18:24:52.676503 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:52.676467    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:52.917576 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:52.917543    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"]
Jul 09 18:24:52.920266 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:52.920248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:24:52.924227 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:52.924202    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 09 18:24:52.924372 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:52.924246    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-zgqn9\""
Jul 09 18:24:52.931272 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:52.931247    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"]
Jul 09 18:24:52.943116 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:52.943082    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:52.984180 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:52.984137    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 09 18:24:52.995147 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:52.995118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2ktq\" (UniqueName: \"kubernetes.io/projected/ce9b0224-d603-46fa-bfa1-851a51d33cdb-kube-api-access-b2ktq\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6\" (UID: \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:24:52.995339 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:52.995169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ce9b0224-d603-46fa-bfa1-851a51d33cdb-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6\" (UID: \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:24:52.995339 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:52.995244    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9b0224-d603-46fa-bfa1-851a51d33cdb-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6\" (UID: \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:24:53.096497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:53.096440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2ktq\" (UniqueName: \"kubernetes.io/projected/ce9b0224-d603-46fa-bfa1-851a51d33cdb-kube-api-access-b2ktq\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6\" (UID: \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:24:53.096723 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:53.096535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ce9b0224-d603-46fa-bfa1-851a51d33cdb-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6\" (UID: \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:24:53.096723 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:53.096578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9b0224-d603-46fa-bfa1-851a51d33cdb-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6\" (UID: \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:24:53.097028 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:53.097004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ce9b0224-d603-46fa-bfa1-851a51d33cdb-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6\" (UID: \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:24:53.099091 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:53.099061    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9b0224-d603-46fa-bfa1-851a51d33cdb-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6\" (UID: \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:24:53.108903 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:53.108867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2ktq\" (UniqueName: \"kubernetes.io/projected/ce9b0224-d603-46fa-bfa1-851a51d33cdb-kube-api-access-b2ktq\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6\" (UID: \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:24:53.230009 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:53.229974    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:24:53.392132 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:53.392109    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"]
Jul 09 18:24:53.395619 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:24:53.395587    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9b0224_d603_46fa_bfa1_851a51d33cdb.slice/crio-46e149d7c71b3c3052f1f1c67b843e8deaab68fc2797a49d0143e3f47c79a092 WatchSource:0}: Error finding container 46e149d7c71b3c3052f1f1c67b843e8deaab68fc2797a49d0143e3f47c79a092: Status 404 returned error can't find the container with id 46e149d7c71b3c3052f1f1c67b843e8deaab68fc2797a49d0143e3f47c79a092
Jul 09 18:24:53.988104 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:53.988059    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6" event={"ID":"ce9b0224-d603-46fa-bfa1-851a51d33cdb","Type":"ContainerStarted","Data":"7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6"}
Jul 09 18:24:53.988682 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:53.988129    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6" event={"ID":"ce9b0224-d603-46fa-bfa1-851a51d33cdb","Type":"ContainerStarted","Data":"46e149d7c71b3c3052f1f1c67b843e8deaab68fc2797a49d0143e3f47c79a092"}
Jul 09 18:24:53.988682 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:53.988187    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:24:54.014961 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:24:54.014905    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6" podStartSLOduration=2.014888851 podStartE2EDuration="2.014888851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:24:54.014000835 +0000 UTC m=+1217.200978630" watchObservedRunningTime="2026-07-09 18:24:54.014888851 +0000 UTC m=+1217.201866661"
Jul 09 18:25:02.983918 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:02.983877    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 09 18:25:12.984055 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:12.984007    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 09 18:25:18.136028 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:18.135990    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:22.984264 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:22.984220    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 09 18:25:24.994761 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:24.994732    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:25:26.552128 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:26.552091    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:26.562623 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:26.562590    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"]
Jul 09 18:25:26.573597 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:25:26.573558    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:25:26.573597 ip-10-0-137-79 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:25:26.573597 ip-10-0-137-79 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 09 18:25:26.573597 ip-10-0-137-79 kubenswrapper[2578]:  > containerID="7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6" cmd=["/bin/sleep","15"]
Jul 09 18:25:26.573958 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:26.573610    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6" message="" err=<
Jul 09 18:25:26.573958 ip-10-0-137-79 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:25:26.573958 ip-10-0-137-79 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 09 18:25:26.573958 ip-10-0-137-79 kubenswrapper[2578]:  >
Jul 09 18:25:26.573958 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:25:26.573622    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:25:26.573958 ip-10-0-137-79 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:25:26.573958 ip-10-0-137-79 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 09 18:25:26.573958 ip-10-0-137-79 kubenswrapper[2578]:  > pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6" podUID="ce9b0224-d603-46fa-bfa1-851a51d33cdb" containerName="main" containerID="cri-o://7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6"
Jul 09 18:25:26.573958 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:26.573662    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6" podUID="ce9b0224-d603-46fa-bfa1-851a51d33cdb" containerName="main" containerID="cri-o://7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6" gracePeriod=60
Jul 09 18:25:26.723813 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:26.723758    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:26.834517 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:26.834486    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:25:26.913131 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:26.913097    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9b0224-d603-46fa-bfa1-851a51d33cdb-tls-certs\") pod \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\" (UID: \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\") "
Jul 09 18:25:26.913312 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:26.913185    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ce9b0224-d603-46fa-bfa1-851a51d33cdb-tokenizer-uds\") pod \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\" (UID: \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\") "
Jul 09 18:25:26.913419 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:26.913398    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9b0224-d603-46fa-bfa1-851a51d33cdb-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ce9b0224-d603-46fa-bfa1-851a51d33cdb" (UID: "ce9b0224-d603-46fa-bfa1-851a51d33cdb"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:26.915169 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:26.915149    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce9b0224-d603-46fa-bfa1-851a51d33cdb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce9b0224-d603-46fa-bfa1-851a51d33cdb" (UID: "ce9b0224-d603-46fa-bfa1-851a51d33cdb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:25:27.014541 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.014501    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2ktq\" (UniqueName: \"kubernetes.io/projected/ce9b0224-d603-46fa-bfa1-851a51d33cdb-kube-api-access-b2ktq\") pod \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\" (UID: \"ce9b0224-d603-46fa-bfa1-851a51d33cdb\") "
Jul 09 18:25:27.014734 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.014722    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9b0224-d603-46fa-bfa1-851a51d33cdb-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:25:27.014791 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.014736    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ce9b0224-d603-46fa-bfa1-851a51d33cdb-tokenizer-uds\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:25:27.016764 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.016731    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9b0224-d603-46fa-bfa1-851a51d33cdb-kube-api-access-b2ktq" (OuterVolumeSpecName: "kube-api-access-b2ktq") pod "ce9b0224-d603-46fa-bfa1-851a51d33cdb" (UID: "ce9b0224-d603-46fa-bfa1-851a51d33cdb"). InnerVolumeSpecName "kube-api-access-b2ktq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:25:27.116113 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.116021    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2ktq\" (UniqueName: \"kubernetes.io/projected/ce9b0224-d603-46fa-bfa1-851a51d33cdb-kube-api-access-b2ktq\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:25:27.150902 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.150868    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce9b0224-d603-46fa-bfa1-851a51d33cdb" containerID="7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6" exitCode=0
Jul 09 18:25:27.151084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.150912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6" event={"ID":"ce9b0224-d603-46fa-bfa1-851a51d33cdb","Type":"ContainerDied","Data":"7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6"}
Jul 09 18:25:27.151084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.150933    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"
Jul 09 18:25:27.151084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.150954    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6" event={"ID":"ce9b0224-d603-46fa-bfa1-851a51d33cdb","Type":"ContainerDied","Data":"46e149d7c71b3c3052f1f1c67b843e8deaab68fc2797a49d0143e3f47c79a092"}
Jul 09 18:25:27.151084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.150974    2578 scope.go:117] "RemoveContainer" containerID="7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6"
Jul 09 18:25:27.159929 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.159908    2578 scope.go:117] "RemoveContainer" containerID="7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6"
Jul 09 18:25:27.160274 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:25:27.160249    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6\": container with ID starting with 7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6 not found: ID does not exist" containerID="7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6"
Jul 09 18:25:27.160388 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.160279    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6"} err="failed to get container status \"7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6\": rpc error: code = NotFound desc = could not find container \"7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6\": container with ID starting with 7f0e02a7330d37d749591f10bd1016d410199f5fabb67a5309ffbbd2206696a6 not found: ID does not exist"
Jul 09 18:25:27.180227 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.180190    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"]
Jul 09 18:25:27.188910 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.188875    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6"]
Jul 09 18:25:27.403669 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:27.403584    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce9b0224-d603-46fa-bfa1-851a51d33cdb" path="/var/lib/kubelet/pods/ce9b0224-d603-46fa-bfa1-851a51d33cdb/volumes"
Jul 09 18:25:32.984496 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:32.984400    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 09 18:25:42.984047 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:42.984001    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 09 18:25:47.059242 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:47.059210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:47.312106 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:47.312018    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:52.983654 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:25:52.983609    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 09 18:26:02.984486 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:02.984443    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 09 18:26:12.984160 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:12.984104    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 09 18:26:18.696686 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.696653    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"]
Jul 09 18:26:18.697227 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.697203    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce9b0224-d603-46fa-bfa1-851a51d33cdb" containerName="main"
Jul 09 18:26:18.697319 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.697232    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce9b0224-d603-46fa-bfa1-851a51d33cdb" containerName="main"
Jul 09 18:26:18.697375 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.697335    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce9b0224-d603-46fa-bfa1-851a51d33cdb" containerName="main"
Jul 09 18:26:18.702533 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.702510    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.705156 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.705130    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 09 18:26:18.705271 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.705199    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:26:18.717453 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.717412    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"]
Jul 09 18:26:18.726353 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.726316    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:18.780223 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.780169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.780223 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.780235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.780491 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.780277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1567e153-102c-482e-8bdb-d8a25b82125d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.780491 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.780302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.780491 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.780320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.780491 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.780349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.780491 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.780364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wt5g\" (UniqueName: \"kubernetes.io/projected/1567e153-102c-482e-8bdb-d8a25b82125d-kube-api-access-7wt5g\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.780491 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.780380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1567e153-102c-482e-8bdb-d8a25b82125d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.881462 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.881409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.881462 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.881452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.881742 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.881501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.881742 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.881527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wt5g\" (UniqueName: \"kubernetes.io/projected/1567e153-102c-482e-8bdb-d8a25b82125d-kube-api-access-7wt5g\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.881742 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.881552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1567e153-102c-482e-8bdb-d8a25b82125d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.881927 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.881762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.881927 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.881878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.881927 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.881921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1567e153-102c-482e-8bdb-d8a25b82125d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.882071 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.881963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.882166 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.882142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.882235 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.882221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.882286 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.882270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1567e153-102c-482e-8bdb-d8a25b82125d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.882452 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.882432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.884238 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.884215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.884519 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.884494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1567e153-102c-482e-8bdb-d8a25b82125d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:18.890891 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:18.890866    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wt5g\" (UniqueName: \"kubernetes.io/projected/1567e153-102c-482e-8bdb-d8a25b82125d-kube-api-access-7wt5g\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:19.014875 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:19.014741    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:19.169041 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:19.168858    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"]
Jul 09 18:26:19.172863 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:26:19.172834    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1567e153_102c_482e_8bdb_d8a25b82125d.slice/crio-504de2fd8853377fb73d01913b3d15d0d7f5c78747a8bb3456ef6c372241ea92 WatchSource:0}: Error finding container 504de2fd8853377fb73d01913b3d15d0d7f5c78747a8bb3456ef6c372241ea92: Status 404 returned error can't find the container with id 504de2fd8853377fb73d01913b3d15d0d7f5c78747a8bb3456ef6c372241ea92
Jul 09 18:26:19.408212 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:19.408130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" event={"ID":"1567e153-102c-482e-8bdb-d8a25b82125d","Type":"ContainerStarted","Data":"fac875990d4d4b6adb95851df114f130aae2b18d0bfec9989d8754da20c46644"}
Jul 09 18:26:19.408212 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:19.408180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" event={"ID":"1567e153-102c-482e-8bdb-d8a25b82125d","Type":"ContainerStarted","Data":"504de2fd8853377fb73d01913b3d15d0d7f5c78747a8bb3456ef6c372241ea92"}
Jul 09 18:26:20.106428 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:20.106389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:22.984140 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:22.984087    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 09 18:26:24.445157 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:24.445110    2578 generic.go:356] "Generic (PLEG): container finished" podID="1567e153-102c-482e-8bdb-d8a25b82125d" containerID="fac875990d4d4b6adb95851df114f130aae2b18d0bfec9989d8754da20c46644" exitCode=0
Jul 09 18:26:24.445714 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:24.445186    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" event={"ID":"1567e153-102c-482e-8bdb-d8a25b82125d","Type":"ContainerDied","Data":"fac875990d4d4b6adb95851df114f130aae2b18d0bfec9989d8754da20c46644"}
Jul 09 18:26:25.452046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:25.452013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" event={"ID":"1567e153-102c-482e-8bdb-d8a25b82125d","Type":"ContainerStarted","Data":"c3118f3ab2b0962a51e57be2122d6b7b29bc6cfde14cf07129c780e0863aa897"}
Jul 09 18:26:25.486715 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:25.486650    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" podStartSLOduration=3.046391419 podStartE2EDuration="7.486629618s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.440238199s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:25.483703413 +0000 UTC m=+1308.670681184" watchObservedRunningTime="2026-07-09 18:26:25.486629618 +0000 UTC m=+1308.673607385"
Jul 09 18:26:26.015537 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:26.015496    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:29.014991 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:29.014953    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:26:29.016604 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:29.016573    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Jul 09 18:26:32.994223 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:32.994187    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:26:33.002550 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:33.002520    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:26:36.395188 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:36.395153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:39.015641 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:39.015594    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Jul 09 18:26:39.056738 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:39.056708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:39.113242 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:39.113209    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:42.916609 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:42.916567    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"]
Jul 09 18:26:42.935409 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:42.935373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:43.043655 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:43.043621    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:49.015655 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:49.015531    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Jul 09 18:26:57.941578 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:57.941533    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main" containerID="cri-o://d6b975ff3d48c37726d6f301cb561ca6ea3f2a76a8383b99a16cb8fe086c484c" gracePeriod=45
Jul 09 18:26:57.995516 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:57.995477    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 09 18:26:58.623997 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.623963    2578 generic.go:356] "Generic (PLEG): container finished" podID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerID="d6b975ff3d48c37726d6f301cb561ca6ea3f2a76a8383b99a16cb8fe086c484c" exitCode=0
Jul 09 18:26:58.624199 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.624001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" event={"ID":"996d2ab5-ede1-4c69-866c-221319c86ccf","Type":"ContainerDied","Data":"d6b975ff3d48c37726d6f301cb561ca6ea3f2a76a8383b99a16cb8fe086c484c"}
Jul 09 18:26:58.710235 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.710210    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:26:58.750375 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.750289    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-model-cache\") pod \"996d2ab5-ede1-4c69-866c-221319c86ccf\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") "
Jul 09 18:26:58.750375 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.750349    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkqf8\" (UniqueName: \"kubernetes.io/projected/996d2ab5-ede1-4c69-866c-221319c86ccf-kube-api-access-jkqf8\") pod \"996d2ab5-ede1-4c69-866c-221319c86ccf\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") "
Jul 09 18:26:58.750375 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.750380    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-kserve-provision-location\") pod \"996d2ab5-ede1-4c69-866c-221319c86ccf\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") "
Jul 09 18:26:58.750727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.750434    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-dshm\") pod \"996d2ab5-ede1-4c69-866c-221319c86ccf\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") "
Jul 09 18:26:58.750727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.750473    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996d2ab5-ede1-4c69-866c-221319c86ccf-tls-certs\") pod \"996d2ab5-ede1-4c69-866c-221319c86ccf\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") "
Jul 09 18:26:58.750727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.750551    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-home\") pod \"996d2ab5-ede1-4c69-866c-221319c86ccf\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") "
Jul 09 18:26:58.750727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.750598    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-tmp-dir\") pod \"996d2ab5-ede1-4c69-866c-221319c86ccf\" (UID: \"996d2ab5-ede1-4c69-866c-221319c86ccf\") "
Jul 09 18:26:58.750727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.750600    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-model-cache" (OuterVolumeSpecName: "model-cache") pod "996d2ab5-ede1-4c69-866c-221319c86ccf" (UID: "996d2ab5-ede1-4c69-866c-221319c86ccf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:58.751161 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.750885    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:26:58.751616 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.751573    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-home" (OuterVolumeSpecName: "home") pod "996d2ab5-ede1-4c69-866c-221319c86ccf" (UID: "996d2ab5-ede1-4c69-866c-221319c86ccf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:58.753339 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.753188    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-dshm" (OuterVolumeSpecName: "dshm") pod "996d2ab5-ede1-4c69-866c-221319c86ccf" (UID: "996d2ab5-ede1-4c69-866c-221319c86ccf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:58.753452 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.753392    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996d2ab5-ede1-4c69-866c-221319c86ccf-kube-api-access-jkqf8" (OuterVolumeSpecName: "kube-api-access-jkqf8") pod "996d2ab5-ede1-4c69-866c-221319c86ccf" (UID: "996d2ab5-ede1-4c69-866c-221319c86ccf"). InnerVolumeSpecName "kube-api-access-jkqf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:58.753893 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.753859    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/996d2ab5-ede1-4c69-866c-221319c86ccf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "996d2ab5-ede1-4c69-866c-221319c86ccf" (UID: "996d2ab5-ede1-4c69-866c-221319c86ccf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:58.770561 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.770506    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "996d2ab5-ede1-4c69-866c-221319c86ccf" (UID: "996d2ab5-ede1-4c69-866c-221319c86ccf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:58.817560 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.817506    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "996d2ab5-ede1-4c69-866c-221319c86ccf" (UID: "996d2ab5-ede1-4c69-866c-221319c86ccf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:58.851234 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.851198    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:26:58.851234 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.851229    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996d2ab5-ede1-4c69-866c-221319c86ccf-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:26:58.851234 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.851239    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:26:58.851461 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.851247    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:26:58.851461 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.851255    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkqf8\" (UniqueName: \"kubernetes.io/projected/996d2ab5-ede1-4c69-866c-221319c86ccf-kube-api-access-jkqf8\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:26:58.851461 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:58.851264    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996d2ab5-ede1-4c69-866c-221319c86ccf-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:26:59.015535 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:59.015430    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Jul 09 18:26:59.630115 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:59.630073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4" event={"ID":"996d2ab5-ede1-4c69-866c-221319c86ccf","Type":"ContainerDied","Data":"a2e89b9516864218cfe2b65f9f59555b32ca06946ff165dd71e05d1a7fa68a2b"}
Jul 09 18:26:59.630115 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:59.630117    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"
Jul 09 18:26:59.630333 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:59.630151    2578 scope.go:117] "RemoveContainer" containerID="d6b975ff3d48c37726d6f301cb561ca6ea3f2a76a8383b99a16cb8fe086c484c"
Jul 09 18:26:59.640539 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:59.640509    2578 scope.go:117] "RemoveContainer" containerID="b9b26adcc18e8efa0110fea99708a7d6bb9ca11c11e498ca6aa9d2d99b15ffc9"
Jul 09 18:26:59.650497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:59.650415    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"]
Jul 09 18:26:59.653087 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:26:59.653063    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4"]
Jul 09 18:27:01.403343 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:01.403304    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" path="/var/lib/kubelet/pods/996d2ab5-ede1-4c69-866c-221319c86ccf/volumes"
Jul 09 18:27:09.015789 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:09.015744    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Jul 09 18:27:19.015690 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:19.015645    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Jul 09 18:27:21.290472 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:21.290436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:27:29.016141 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:29.016083    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Jul 09 18:27:31.785275 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:31.785241    2578 generic.go:356] "Generic (PLEG): container finished" podID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" containerID="e03ddd03cf6f20377a900c84467f0dcffa72bd772b8d6c968c7587e17584330e" exitCode=0
Jul 09 18:27:31.785727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:31.785310    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs" event={"ID":"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d","Type":"ContainerDied","Data":"e03ddd03cf6f20377a900c84467f0dcffa72bd772b8d6c968c7587e17584330e"}
Jul 09 18:27:31.785772 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:31.785748    2578 scope.go:117] "RemoveContainer" containerID="e03ddd03cf6f20377a900c84467f0dcffa72bd772b8d6c968c7587e17584330e"
Jul 09 18:27:31.786757 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:31.786729    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:27:31.786867 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:31.786825    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:27:31.786867 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:31.786858    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:27:32.792135 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:32.792099    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs" event={"ID":"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d","Type":"ContainerStarted","Data":"8b89b1462854051443c9337dc19247a21c69730666ca6d4878788fcd5e26ec4d"}
Jul 09 18:27:33.407290 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:33.407252    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:27:33.407539 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:33.407513    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs" podUID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Jul 09 18:27:34.408047 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:34.407989    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs" podUID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Jul 09 18:27:35.407953 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:35.407906    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs" podUID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Jul 09 18:27:36.407462 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:36.407417    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs" podUID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Jul 09 18:27:37.407595 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:37.407553    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs" podUID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Jul 09 18:27:38.432419 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:38.432375    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:27:38.432952 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:38.432755    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:27:38.433993 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:38.433969    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:27:39.015553 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:39.015503    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Jul 09 18:27:41.868233 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:41.868197    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:27:49.025966 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:49.025882    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:27:49.034119 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:27:49.034084    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:28:22.598711 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.598676    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"]
Jul 09 18:28:22.599183 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.599039    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main"
Jul 09 18:28:22.599183 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.599052    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main"
Jul 09 18:28:22.599183 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.599074    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="storage-initializer"
Jul 09 18:28:22.599183 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.599080    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="storage-initializer"
Jul 09 18:28:22.599183 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.599129    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="996d2ab5-ede1-4c69-866c-221319c86ccf" containerName="main"
Jul 09 18:28:22.604739 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.604717    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.609686 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.609660    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 09 18:28:22.643276 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.643241    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"]
Jul 09 18:28:22.682165 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.682124    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:28:22.702318 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.702270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.702318 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.702322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86wr\" (UniqueName: \"kubernetes.io/projected/39530515-a4e1-4a98-aea8-85d62fb30a9f-kube-api-access-d86wr\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.702541 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.702350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39530515-a4e1-4a98-aea8-85d62fb30a9f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.702541 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.702418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.702541 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.702447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.702541 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.702504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.702717 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.702576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.803558 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.803519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.803769 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.803565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.803769 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.803611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.803769 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.803648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.803769 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.803715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.803769 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.803765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86wr\" (UniqueName: \"kubernetes.io/projected/39530515-a4e1-4a98-aea8-85d62fb30a9f-kube-api-access-d86wr\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.804091 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.803823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39530515-a4e1-4a98-aea8-85d62fb30a9f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.804091 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.804026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.804091 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.804065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.804250 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.804111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.804250 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.804161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.806005 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.805985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.806283 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.806264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39530515-a4e1-4a98-aea8-85d62fb30a9f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.813620 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.813593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86wr\" (UniqueName: \"kubernetes.io/projected/39530515-a4e1-4a98-aea8-85d62fb30a9f-kube-api-access-d86wr\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:22.915909 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:22.915786    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:23.039210 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:23.039126    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:28:23.088529 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:23.088493    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"]
Jul 09 18:28:23.089520 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:28:23.089494    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39530515_a4e1_4a98_aea8_85d62fb30a9f.slice/crio-0c7b8ab822d6a6e1066cf160f919a3066c9787ad687c7e1b0c54ff5eeff6e71f WatchSource:0}: Error finding container 0c7b8ab822d6a6e1066cf160f919a3066c9787ad687c7e1b0c54ff5eeff6e71f: Status 404 returned error can't find the container with id 0c7b8ab822d6a6e1066cf160f919a3066c9787ad687c7e1b0c54ff5eeff6e71f
Jul 09 18:28:24.039896 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:24.039856    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" event={"ID":"39530515-a4e1-4a98-aea8-85d62fb30a9f","Type":"ContainerStarted","Data":"2174b34e62db280f402d9438efce72956e0e63abe07c7e5bcbf1283b13ecaad8"}
Jul 09 18:28:24.040358 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:24.039917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" event={"ID":"39530515-a4e1-4a98-aea8-85d62fb30a9f","Type":"ContainerStarted","Data":"0c7b8ab822d6a6e1066cf160f919a3066c9787ad687c7e1b0c54ff5eeff6e71f"}
Jul 09 18:28:27.060323 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:27.060277    2578 generic.go:356] "Generic (PLEG): container finished" podID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerID="2174b34e62db280f402d9438efce72956e0e63abe07c7e5bcbf1283b13ecaad8" exitCode=0
Jul 09 18:28:27.060853 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:27.060352    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" event={"ID":"39530515-a4e1-4a98-aea8-85d62fb30a9f","Type":"ContainerDied","Data":"2174b34e62db280f402d9438efce72956e0e63abe07c7e5bcbf1283b13ecaad8"}
Jul 09 18:28:27.224936 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:27.224889    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:28:27.269540 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:27.269505    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"]
Jul 09 18:28:27.348849 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:27.348682    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:28:28.067299 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:28.067260    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" event={"ID":"39530515-a4e1-4a98-aea8-85d62fb30a9f","Type":"ContainerStarted","Data":"ce902da1db3751ca011d3a596ba151b721942d768a1e9d4645fa10d799eaa10e"}
Jul 09 18:28:28.103241 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:28.103170    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podStartSLOduration=2.5649406580000003 podStartE2EDuration="6.103147292s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.538206634s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:28:28.101722263 +0000 UTC m=+1431.288700053" watchObservedRunningTime="2026-07-09 18:28:28.103147292 +0000 UTC m=+1431.290125062"
Jul 09 18:28:28.916453 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:28.916401    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:31.135825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:31.135773    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:28:32.916698 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:32.916662    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:28:32.918245 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:32.918218    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 09 18:28:42.293090 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:42.293018    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="main" containerID="cri-o://c3118f3ab2b0962a51e57be2122d6b7b29bc6cfde14cf07129c780e0863aa897" gracePeriod=45
Jul 09 18:28:42.789857 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:42.789824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:28:42.917019 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:42.916975    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 09 18:28:43.052982 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.052953    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:28:43.096990 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.096958    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-model-cache\") pod \"1567e153-102c-482e-8bdb-d8a25b82125d\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") "
Jul 09 18:28:43.097177 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.097007    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1567e153-102c-482e-8bdb-d8a25b82125d-cabundle-cert\") pod \"1567e153-102c-482e-8bdb-d8a25b82125d\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") "
Jul 09 18:28:43.097177 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.097040    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-tmp-dir\") pod \"1567e153-102c-482e-8bdb-d8a25b82125d\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") "
Jul 09 18:28:43.097177 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.097082    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-home\") pod \"1567e153-102c-482e-8bdb-d8a25b82125d\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") "
Jul 09 18:28:43.097177 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.097114    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wt5g\" (UniqueName: \"kubernetes.io/projected/1567e153-102c-482e-8bdb-d8a25b82125d-kube-api-access-7wt5g\") pod \"1567e153-102c-482e-8bdb-d8a25b82125d\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") "
Jul 09 18:28:43.097177 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.097140    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1567e153-102c-482e-8bdb-d8a25b82125d-tls-certs\") pod \"1567e153-102c-482e-8bdb-d8a25b82125d\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") "
Jul 09 18:28:43.097458 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.097181    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-dshm\") pod \"1567e153-102c-482e-8bdb-d8a25b82125d\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") "
Jul 09 18:28:43.097458 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.097211    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-kserve-provision-location\") pod \"1567e153-102c-482e-8bdb-d8a25b82125d\" (UID: \"1567e153-102c-482e-8bdb-d8a25b82125d\") "
Jul 09 18:28:43.097458 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.097261    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-model-cache" (OuterVolumeSpecName: "model-cache") pod "1567e153-102c-482e-8bdb-d8a25b82125d" (UID: "1567e153-102c-482e-8bdb-d8a25b82125d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:43.097617 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.097469    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:28:43.097617 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.097486    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1567e153-102c-482e-8bdb-d8a25b82125d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1567e153-102c-482e-8bdb-d8a25b82125d" (UID: "1567e153-102c-482e-8bdb-d8a25b82125d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:43.097887 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.097847    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-home" (OuterVolumeSpecName: "home") pod "1567e153-102c-482e-8bdb-d8a25b82125d" (UID: "1567e153-102c-482e-8bdb-d8a25b82125d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:43.100148 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.100081    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-dshm" (OuterVolumeSpecName: "dshm") pod "1567e153-102c-482e-8bdb-d8a25b82125d" (UID: "1567e153-102c-482e-8bdb-d8a25b82125d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:43.100148 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.100084    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1567e153-102c-482e-8bdb-d8a25b82125d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1567e153-102c-482e-8bdb-d8a25b82125d" (UID: "1567e153-102c-482e-8bdb-d8a25b82125d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:28:43.100285 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.100179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1567e153-102c-482e-8bdb-d8a25b82125d-kube-api-access-7wt5g" (OuterVolumeSpecName: "kube-api-access-7wt5g") pod "1567e153-102c-482e-8bdb-d8a25b82125d" (UID: "1567e153-102c-482e-8bdb-d8a25b82125d"). InnerVolumeSpecName "kube-api-access-7wt5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:28:43.115354 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.115295    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1567e153-102c-482e-8bdb-d8a25b82125d" (UID: "1567e153-102c-482e-8bdb-d8a25b82125d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:43.145667 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.145627    2578 generic.go:356] "Generic (PLEG): container finished" podID="1567e153-102c-482e-8bdb-d8a25b82125d" containerID="c3118f3ab2b0962a51e57be2122d6b7b29bc6cfde14cf07129c780e0863aa897" exitCode=0
Jul 09 18:28:43.145864 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.145669    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" event={"ID":"1567e153-102c-482e-8bdb-d8a25b82125d","Type":"ContainerDied","Data":"c3118f3ab2b0962a51e57be2122d6b7b29bc6cfde14cf07129c780e0863aa897"}
Jul 09 18:28:43.145864 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.145705    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"
Jul 09 18:28:43.145864 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.145725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" event={"ID":"1567e153-102c-482e-8bdb-d8a25b82125d","Type":"ContainerDied","Data":"504de2fd8853377fb73d01913b3d15d0d7f5c78747a8bb3456ef6c372241ea92"}
Jul 09 18:28:43.145864 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.145752    2578 scope.go:117] "RemoveContainer" containerID="c3118f3ab2b0962a51e57be2122d6b7b29bc6cfde14cf07129c780e0863aa897"
Jul 09 18:28:43.155362 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.155336    2578 scope.go:117] "RemoveContainer" containerID="fac875990d4d4b6adb95851df114f130aae2b18d0bfec9989d8754da20c46644"
Jul 09 18:28:43.164084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.164063    2578 scope.go:117] "RemoveContainer" containerID="c3118f3ab2b0962a51e57be2122d6b7b29bc6cfde14cf07129c780e0863aa897"
Jul 09 18:28:43.164433 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:28:43.164407    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3118f3ab2b0962a51e57be2122d6b7b29bc6cfde14cf07129c780e0863aa897\": container with ID starting with c3118f3ab2b0962a51e57be2122d6b7b29bc6cfde14cf07129c780e0863aa897 not found: ID does not exist" containerID="c3118f3ab2b0962a51e57be2122d6b7b29bc6cfde14cf07129c780e0863aa897"
Jul 09 18:28:43.164505 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.164448    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3118f3ab2b0962a51e57be2122d6b7b29bc6cfde14cf07129c780e0863aa897"} err="failed to get container status \"c3118f3ab2b0962a51e57be2122d6b7b29bc6cfde14cf07129c780e0863aa897\": rpc error: code = NotFound desc = could not find container \"c3118f3ab2b0962a51e57be2122d6b7b29bc6cfde14cf07129c780e0863aa897\": container with ID starting with c3118f3ab2b0962a51e57be2122d6b7b29bc6cfde14cf07129c780e0863aa897 not found: ID does not exist"
Jul 09 18:28:43.164505 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.164477    2578 scope.go:117] "RemoveContainer" containerID="fac875990d4d4b6adb95851df114f130aae2b18d0bfec9989d8754da20c46644"
Jul 09 18:28:43.164773 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:28:43.164752    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac875990d4d4b6adb95851df114f130aae2b18d0bfec9989d8754da20c46644\": container with ID starting with fac875990d4d4b6adb95851df114f130aae2b18d0bfec9989d8754da20c46644 not found: ID does not exist" containerID="fac875990d4d4b6adb95851df114f130aae2b18d0bfec9989d8754da20c46644"
Jul 09 18:28:43.164873 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.164782    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac875990d4d4b6adb95851df114f130aae2b18d0bfec9989d8754da20c46644"} err="failed to get container status \"fac875990d4d4b6adb95851df114f130aae2b18d0bfec9989d8754da20c46644\": rpc error: code = NotFound desc = could not find container \"fac875990d4d4b6adb95851df114f130aae2b18d0bfec9989d8754da20c46644\": container with ID starting with fac875990d4d4b6adb95851df114f130aae2b18d0bfec9989d8754da20c46644 not found: ID does not exist"
Jul 09 18:28:43.165498 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.165468    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1567e153-102c-482e-8bdb-d8a25b82125d" (UID: "1567e153-102c-482e-8bdb-d8a25b82125d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:43.198485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.198442    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:28:43.198485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.198479    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1567e153-102c-482e-8bdb-d8a25b82125d-cabundle-cert\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:28:43.198485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.198491    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:28:43.198728 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.198501    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:28:43.198728 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.198513    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wt5g\" (UniqueName: \"kubernetes.io/projected/1567e153-102c-482e-8bdb-d8a25b82125d-kube-api-access-7wt5g\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:28:43.198728 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.198523    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1567e153-102c-482e-8bdb-d8a25b82125d-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:28:43.198728 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.198535    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1567e153-102c-482e-8bdb-d8a25b82125d-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:28:43.464075 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.464042    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"]
Jul 09 18:28:43.469427 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:43.469396    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt"]
Jul 09 18:28:44.026524 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:44.026473    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": context deadline exceeded"
Jul 09 18:28:45.404234 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:45.404200    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" path="/var/lib/kubelet/pods/1567e153-102c-482e-8bdb-d8a25b82125d/volumes"
Jul 09 18:28:52.916410 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:28:52.916366    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 09 18:29:02.916841 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:02.916777    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 09 18:29:06.909018 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:06.908984    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"]
Jul 09 18:29:06.909400 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:06.909289    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="main"
Jul 09 18:29:06.909400 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:06.909299    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="main"
Jul 09 18:29:06.909400 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:06.909322    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="storage-initializer"
Jul 09 18:29:06.909400 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:06.909328    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="storage-initializer"
Jul 09 18:29:06.909400 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:06.909377    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1567e153-102c-482e-8bdb-d8a25b82125d" containerName="main"
Jul 09 18:29:06.911256 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:06.911238    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:06.914179 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:06.914154    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:29:06.914179 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:06.914173    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 09 18:29:06.928506 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:06.928473    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"]
Jul 09 18:29:06.934057 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:06.934022    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:07.028725 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.028687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.028725 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.028730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc51b02-e94c-4377-bb76-17410eb5858b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.028978 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.028758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc51b02-e94c-4377-bb76-17410eb5858b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.028978 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.028835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.028978 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.028859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.028978 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.028898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.029106 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.028994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sppvc\" (UniqueName: \"kubernetes.io/projected/efc51b02-e94c-4377-bb76-17410eb5858b-kube-api-access-sppvc\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.029106 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.029074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.129849 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.129771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.130046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.129868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc51b02-e94c-4377-bb76-17410eb5858b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.130046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.129905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc51b02-e94c-4377-bb76-17410eb5858b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.130046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.129948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.130046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.129975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.130046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.130037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.130331 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.130074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppvc\" (UniqueName: \"kubernetes.io/projected/efc51b02-e94c-4377-bb76-17410eb5858b-kube-api-access-sppvc\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.130331 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.130141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.130331 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.130233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.130482 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.130463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.130544 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.130480    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.130544 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.130502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.131031 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.131008    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc51b02-e94c-4377-bb76-17410eb5858b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.132362 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.132331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.132459 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.132433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc51b02-e94c-4377-bb76-17410eb5858b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.141587 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.141560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sppvc\" (UniqueName: \"kubernetes.io/projected/efc51b02-e94c-4377-bb76-17410eb5858b-kube-api-access-sppvc\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.221856 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.221781    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:07.375136 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.375089    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"]
Jul 09 18:29:07.404636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:07.404599    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:08.267654 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:08.267600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" event={"ID":"efc51b02-e94c-4377-bb76-17410eb5858b","Type":"ContainerStarted","Data":"e2e854643f72ec185ece7678b0a9cf96e3fb59be63ad2e786e076a59f2c8165b"}
Jul 09 18:29:08.268447 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:08.267725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" event={"ID":"efc51b02-e94c-4377-bb76-17410eb5858b","Type":"ContainerStarted","Data":"7730ad231223ba7133995b66ecf777b2d0ced5489b7e3ca89d5cfcba5329f5ba"}
Jul 09 18:29:11.284410 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:11.284370    2578 generic.go:356] "Generic (PLEG): container finished" podID="efc51b02-e94c-4377-bb76-17410eb5858b" containerID="e2e854643f72ec185ece7678b0a9cf96e3fb59be63ad2e786e076a59f2c8165b" exitCode=0
Jul 09 18:29:11.284842 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:11.284439    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" event={"ID":"efc51b02-e94c-4377-bb76-17410eb5858b","Type":"ContainerDied","Data":"e2e854643f72ec185ece7678b0a9cf96e3fb59be63ad2e786e076a59f2c8165b"}
Jul 09 18:29:12.292461 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:12.292426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" event={"ID":"efc51b02-e94c-4377-bb76-17410eb5858b","Type":"ContainerStarted","Data":"d64e02bcd048fad8fed70df04d3874ad58fa37e7e7c36d5c47ef2c3fff9c51fb"}
Jul 09 18:29:12.319066 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:12.318998    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" podStartSLOduration=2.7400772509999998 podStartE2EDuration="6.318975417s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.578898166s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:29:12.316690883 +0000 UTC m=+1475.503668681" watchObservedRunningTime="2026-07-09 18:29:12.318975417 +0000 UTC m=+1475.505953186"
Jul 09 18:29:12.917251 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:12.917202    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 09 18:29:13.222314 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:13.222275    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:17.222784 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:17.222746    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:29:17.224525 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:17.224494    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 09 18:29:22.916513 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:22.916461    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 09 18:29:27.223287 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:27.223233    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 09 18:29:32.335825 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:32.335770    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:32.917122 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:32.917071    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 09 18:29:37.223059 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:37.223020    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 09 18:29:37.348058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:37.348027    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:29:37.352211 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:37.352177    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:29:37.358198 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:37.358170    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:29:37.362009 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:37.361983    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:29:42.917158 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:42.917097    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 09 18:29:47.222980 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:47.222933    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 09 18:29:52.917273 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:52.917223    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 09 18:29:57.222406 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:29:57.222358    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 09 18:30:02.916981 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:02.916879    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 09 18:30:07.222659 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:07.222615    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 09 18:30:12.917354 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:12.917294    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 09 18:30:17.223185 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:17.223130    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 09 18:30:22.916525 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:22.916473    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 09 18:30:27.222673 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:27.222625    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 09 18:30:32.927058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:32.927025    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:30:32.935580 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:32.935555    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:30:37.232641 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:37.232607    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:30:37.241033 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:37.241000    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:30:38.679343 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:38.679310    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"]
Jul 09 18:30:38.686524 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:38.686495    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:30:38.779535 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:38.779501    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:30:47.153506 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:47.153467    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:30:47.306349 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:47.306312    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:30:51.290693 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:51.290663    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"]
Jul 09 18:30:51.296197 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:51.296169    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:30:51.392002 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:51.391967    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:30:53.705504 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:53.705462    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main" containerID="cri-o://ce902da1db3751ca011d3a596ba151b721942d768a1e9d4645fa10d799eaa10e" gracePeriod=45
Jul 09 18:30:53.928248 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:53.928203    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 09 18:30:54.467367 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.467339    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:30:54.541616 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.541521    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-home\") pod \"39530515-a4e1-4a98-aea8-85d62fb30a9f\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") "
Jul 09 18:30:54.541616 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.541566    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-model-cache\") pod \"39530515-a4e1-4a98-aea8-85d62fb30a9f\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") "
Jul 09 18:30:54.541616 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.541604    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39530515-a4e1-4a98-aea8-85d62fb30a9f-tls-certs\") pod \"39530515-a4e1-4a98-aea8-85d62fb30a9f\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") "
Jul 09 18:30:54.541892 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.541635    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-kserve-provision-location\") pod \"39530515-a4e1-4a98-aea8-85d62fb30a9f\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") "
Jul 09 18:30:54.541892 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.541668    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-dshm\") pod \"39530515-a4e1-4a98-aea8-85d62fb30a9f\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") "
Jul 09 18:30:54.541892 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.541710    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-tmp-dir\") pod \"39530515-a4e1-4a98-aea8-85d62fb30a9f\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") "
Jul 09 18:30:54.541892 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.541839    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d86wr\" (UniqueName: \"kubernetes.io/projected/39530515-a4e1-4a98-aea8-85d62fb30a9f-kube-api-access-d86wr\") pod \"39530515-a4e1-4a98-aea8-85d62fb30a9f\" (UID: \"39530515-a4e1-4a98-aea8-85d62fb30a9f\") "
Jul 09 18:30:54.542092 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.541890    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "39530515-a4e1-4a98-aea8-85d62fb30a9f" (UID: "39530515-a4e1-4a98-aea8-85d62fb30a9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:54.542092 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.542071    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:30:54.542328 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.542289    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-home" (OuterVolumeSpecName: "home") pod "39530515-a4e1-4a98-aea8-85d62fb30a9f" (UID: "39530515-a4e1-4a98-aea8-85d62fb30a9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:54.544446 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.544412    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-dshm" (OuterVolumeSpecName: "dshm") pod "39530515-a4e1-4a98-aea8-85d62fb30a9f" (UID: "39530515-a4e1-4a98-aea8-85d62fb30a9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:54.544611 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.544594    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39530515-a4e1-4a98-aea8-85d62fb30a9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39530515-a4e1-4a98-aea8-85d62fb30a9f" (UID: "39530515-a4e1-4a98-aea8-85d62fb30a9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:30:54.544689 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.544666    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39530515-a4e1-4a98-aea8-85d62fb30a9f-kube-api-access-d86wr" (OuterVolumeSpecName: "kube-api-access-d86wr") pod "39530515-a4e1-4a98-aea8-85d62fb30a9f" (UID: "39530515-a4e1-4a98-aea8-85d62fb30a9f"). InnerVolumeSpecName "kube-api-access-d86wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:30:54.556618 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.556577    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39530515-a4e1-4a98-aea8-85d62fb30a9f" (UID: "39530515-a4e1-4a98-aea8-85d62fb30a9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:54.603696 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.603646    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39530515-a4e1-4a98-aea8-85d62fb30a9f" (UID: "39530515-a4e1-4a98-aea8-85d62fb30a9f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:54.643501 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.643461    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d86wr\" (UniqueName: \"kubernetes.io/projected/39530515-a4e1-4a98-aea8-85d62fb30a9f-kube-api-access-d86wr\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:30:54.643501 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.643494    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:30:54.643501 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.643505    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39530515-a4e1-4a98-aea8-85d62fb30a9f-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:30:54.643850 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.643517    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:30:54.643850 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.643525    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:30:54.643850 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.643532    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39530515-a4e1-4a98-aea8-85d62fb30a9f-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:30:54.816711 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.816627    2578 generic.go:356] "Generic (PLEG): container finished" podID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerID="ce902da1db3751ca011d3a596ba151b721942d768a1e9d4645fa10d799eaa10e" exitCode=0
Jul 09 18:30:54.816711 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.816678    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"
Jul 09 18:30:54.817196 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.816714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" event={"ID":"39530515-a4e1-4a98-aea8-85d62fb30a9f","Type":"ContainerDied","Data":"ce902da1db3751ca011d3a596ba151b721942d768a1e9d4645fa10d799eaa10e"}
Jul 09 18:30:54.817196 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.816770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd" event={"ID":"39530515-a4e1-4a98-aea8-85d62fb30a9f","Type":"ContainerDied","Data":"0c7b8ab822d6a6e1066cf160f919a3066c9787ad687c7e1b0c54ff5eeff6e71f"}
Jul 09 18:30:54.817196 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.816817    2578 scope.go:117] "RemoveContainer" containerID="ce902da1db3751ca011d3a596ba151b721942d768a1e9d4645fa10d799eaa10e"
Jul 09 18:30:54.826152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.826129    2578 scope.go:117] "RemoveContainer" containerID="2174b34e62db280f402d9438efce72956e0e63abe07c7e5bcbf1283b13ecaad8"
Jul 09 18:30:54.834438 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.834419    2578 scope.go:117] "RemoveContainer" containerID="ce902da1db3751ca011d3a596ba151b721942d768a1e9d4645fa10d799eaa10e"
Jul 09 18:30:54.834710 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:30:54.834690    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce902da1db3751ca011d3a596ba151b721942d768a1e9d4645fa10d799eaa10e\": container with ID starting with ce902da1db3751ca011d3a596ba151b721942d768a1e9d4645fa10d799eaa10e not found: ID does not exist" containerID="ce902da1db3751ca011d3a596ba151b721942d768a1e9d4645fa10d799eaa10e"
Jul 09 18:30:54.834786 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.834724    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce902da1db3751ca011d3a596ba151b721942d768a1e9d4645fa10d799eaa10e"} err="failed to get container status \"ce902da1db3751ca011d3a596ba151b721942d768a1e9d4645fa10d799eaa10e\": rpc error: code = NotFound desc = could not find container \"ce902da1db3751ca011d3a596ba151b721942d768a1e9d4645fa10d799eaa10e\": container with ID starting with ce902da1db3751ca011d3a596ba151b721942d768a1e9d4645fa10d799eaa10e not found: ID does not exist"
Jul 09 18:30:54.834786 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.834761    2578 scope.go:117] "RemoveContainer" containerID="2174b34e62db280f402d9438efce72956e0e63abe07c7e5bcbf1283b13ecaad8"
Jul 09 18:30:54.835052 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:30:54.835035    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2174b34e62db280f402d9438efce72956e0e63abe07c7e5bcbf1283b13ecaad8\": container with ID starting with 2174b34e62db280f402d9438efce72956e0e63abe07c7e5bcbf1283b13ecaad8 not found: ID does not exist" containerID="2174b34e62db280f402d9438efce72956e0e63abe07c7e5bcbf1283b13ecaad8"
Jul 09 18:30:54.835099 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.835059    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2174b34e62db280f402d9438efce72956e0e63abe07c7e5bcbf1283b13ecaad8"} err="failed to get container status \"2174b34e62db280f402d9438efce72956e0e63abe07c7e5bcbf1283b13ecaad8\": rpc error: code = NotFound desc = could not find container \"2174b34e62db280f402d9438efce72956e0e63abe07c7e5bcbf1283b13ecaad8\": container with ID starting with 2174b34e62db280f402d9438efce72956e0e63abe07c7e5bcbf1283b13ecaad8 not found: ID does not exist"
Jul 09 18:30:54.841286 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.841261    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"]
Jul 09 18:30:54.846195 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:54.846171    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd"]
Jul 09 18:30:55.403026 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:30:55.402991    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" path="/var/lib/kubelet/pods/39530515-a4e1-4a98-aea8-85d62fb30a9f/volumes"
Jul 09 18:31:04.296740 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.296703    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"]
Jul 09 18:31:04.297174 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.297072    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main"
Jul 09 18:31:04.297174 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.297087    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main"
Jul 09 18:31:04.297174 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.297108    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="storage-initializer"
Jul 09 18:31:04.297174 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.297118    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="storage-initializer"
Jul 09 18:31:04.297311 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.297184    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="39530515-a4e1-4a98-aea8-85d62fb30a9f" containerName="main"
Jul 09 18:31:04.301879 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.301858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.304836 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.304809    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 09 18:31:04.312514 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.312485    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"]
Jul 09 18:31:04.344018 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.343977    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:31:04.432531 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.432486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-tls-certs\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.432531 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.432532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-model-cache\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.432752 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.432564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-home\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.432752 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.432603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.432752 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.432645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-tmp-dir\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.432752 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.432666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-dshm\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.432908 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.432761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftbrf\" (UniqueName: \"kubernetes.io/projected/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-kube-api-access-ftbrf\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.466411 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.466378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:31:04.533747 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.533712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-tls-certs\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.533747 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.533753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-model-cache\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.534018 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.533780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-home\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.534018 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.533857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.534018 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.533881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-tmp-dir\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.534018 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.533910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-dshm\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.534018 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.533983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftbrf\" (UniqueName: \"kubernetes.io/projected/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-kube-api-access-ftbrf\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.534278 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.534225    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-model-cache\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.534337 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.534284    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.534464 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.534428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-home\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.534604 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.534571    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-tmp-dir\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.536361 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.536335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-dshm\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.537135 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.537109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-tls-certs\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.555591 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.555520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftbrf\" (UniqueName: \"kubernetes.io/projected/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-kube-api-access-ftbrf\") pod \"custom-route-timeout-test-kserve-7c7d7b5864-rt6fw\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.612993 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.612949    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:04.614609 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.614585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:31:04.771726 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.771686    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"]
Jul 09 18:31:04.775929 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:31:04.775900    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbca4af_b0aa_4dd6_8cff_64b352bf184b.slice/crio-250326b4f6272cd02f6a9f10a132713da0b3b3dff269a4f7c363703f39bd0129 WatchSource:0}: Error finding container 250326b4f6272cd02f6a9f10a132713da0b3b3dff269a4f7c363703f39bd0129: Status 404 returned error can't find the container with id 250326b4f6272cd02f6a9f10a132713da0b3b3dff269a4f7c363703f39bd0129
Jul 09 18:31:04.866894 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.866848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" event={"ID":"ddbca4af-b0aa-4dd6-8cff-64b352bf184b","Type":"ContainerStarted","Data":"d4ae11d56810243b8042f84dcfb7cfe4219bb8872bbe6be5e3a831d7d1a13ad2"}
Jul 09 18:31:04.867081 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:04.866916    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" event={"ID":"ddbca4af-b0aa-4dd6-8cff-64b352bf184b","Type":"ContainerStarted","Data":"250326b4f6272cd02f6a9f10a132713da0b3b3dff269a4f7c363703f39bd0129"}
Jul 09 18:31:06.315297 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:06.315258    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" containerName="main" containerID="cri-o://d64e02bcd048fad8fed70df04d3874ad58fa37e7e7c36d5c47ef2c3fff9c51fb" gracePeriod=45
Jul 09 18:31:06.882734 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:06.882703    2578 generic.go:356] "Generic (PLEG): container finished" podID="efc51b02-e94c-4377-bb76-17410eb5858b" containerID="d64e02bcd048fad8fed70df04d3874ad58fa37e7e7c36d5c47ef2c3fff9c51fb" exitCode=0
Jul 09 18:31:06.882965 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:06.882777    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" event={"ID":"efc51b02-e94c-4377-bb76-17410eb5858b","Type":"ContainerDied","Data":"d64e02bcd048fad8fed70df04d3874ad58fa37e7e7c36d5c47ef2c3fff9c51fb"}
Jul 09 18:31:06.997572 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:06.997543    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:31:07.059553 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.059453    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-dshm\") pod \"efc51b02-e94c-4377-bb76-17410eb5858b\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") "
Jul 09 18:31:07.059553 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.059500    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-kserve-provision-location\") pod \"efc51b02-e94c-4377-bb76-17410eb5858b\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") "
Jul 09 18:31:07.059553 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.059543    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-model-cache\") pod \"efc51b02-e94c-4377-bb76-17410eb5858b\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") "
Jul 09 18:31:07.059875 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.059602    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sppvc\" (UniqueName: \"kubernetes.io/projected/efc51b02-e94c-4377-bb76-17410eb5858b-kube-api-access-sppvc\") pod \"efc51b02-e94c-4377-bb76-17410eb5858b\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") "
Jul 09 18:31:07.059875 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.059628    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-home\") pod \"efc51b02-e94c-4377-bb76-17410eb5858b\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") "
Jul 09 18:31:07.059875 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.059666    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-tmp-dir\") pod \"efc51b02-e94c-4377-bb76-17410eb5858b\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") "
Jul 09 18:31:07.059875 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.059692    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc51b02-e94c-4377-bb76-17410eb5858b-cabundle-cert\") pod \"efc51b02-e94c-4377-bb76-17410eb5858b\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") "
Jul 09 18:31:07.059875 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.059719    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc51b02-e94c-4377-bb76-17410eb5858b-tls-certs\") pod \"efc51b02-e94c-4377-bb76-17410eb5858b\" (UID: \"efc51b02-e94c-4377-bb76-17410eb5858b\") "
Jul 09 18:31:07.060547 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.060129    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efc51b02-e94c-4377-bb76-17410eb5858b" (UID: "efc51b02-e94c-4377-bb76-17410eb5858b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:07.060547 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.060410    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-model-cache" (OuterVolumeSpecName: "model-cache") pod "efc51b02-e94c-4377-bb76-17410eb5858b" (UID: "efc51b02-e94c-4377-bb76-17410eb5858b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:07.060547 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.060446    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc51b02-e94c-4377-bb76-17410eb5858b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "efc51b02-e94c-4377-bb76-17410eb5858b" (UID: "efc51b02-e94c-4377-bb76-17410eb5858b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:31:07.060787 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.060597    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-home" (OuterVolumeSpecName: "home") pod "efc51b02-e94c-4377-bb76-17410eb5858b" (UID: "efc51b02-e94c-4377-bb76-17410eb5858b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:07.061952 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.061930    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-dshm" (OuterVolumeSpecName: "dshm") pod "efc51b02-e94c-4377-bb76-17410eb5858b" (UID: "efc51b02-e94c-4377-bb76-17410eb5858b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:07.062082 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.062063    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc51b02-e94c-4377-bb76-17410eb5858b-kube-api-access-sppvc" (OuterVolumeSpecName: "kube-api-access-sppvc") pod "efc51b02-e94c-4377-bb76-17410eb5858b" (UID: "efc51b02-e94c-4377-bb76-17410eb5858b"). InnerVolumeSpecName "kube-api-access-sppvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:31:07.062627 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.062595    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc51b02-e94c-4377-bb76-17410eb5858b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efc51b02-e94c-4377-bb76-17410eb5858b" (UID: "efc51b02-e94c-4377-bb76-17410eb5858b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:31:07.126011 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.125963    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "efc51b02-e94c-4377-bb76-17410eb5858b" (UID: "efc51b02-e94c-4377-bb76-17410eb5858b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:07.160627 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.160589    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:31:07.160627 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.160619    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc51b02-e94c-4377-bb76-17410eb5858b-cabundle-cert\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:31:07.160627 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.160629    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc51b02-e94c-4377-bb76-17410eb5858b-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:31:07.160627 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.160636    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:31:07.160934 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.160647    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:31:07.160934 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.160659    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:31:07.160934 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.160668    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sppvc\" (UniqueName: \"kubernetes.io/projected/efc51b02-e94c-4377-bb76-17410eb5858b-kube-api-access-sppvc\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:31:07.160934 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.160677    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc51b02-e94c-4377-bb76-17410eb5858b-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:31:07.889573 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.889536    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs" event={"ID":"efc51b02-e94c-4377-bb76-17410eb5858b","Type":"ContainerDied","Data":"7730ad231223ba7133995b66ecf777b2d0ced5489b7e3ca89d5cfcba5329f5ba"}
Jul 09 18:31:07.890023 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.889606    2578 scope.go:117] "RemoveContainer" containerID="d64e02bcd048fad8fed70df04d3874ad58fa37e7e7c36d5c47ef2c3fff9c51fb"
Jul 09 18:31:07.890023 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.889555    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"
Jul 09 18:31:07.898187 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.898163    2578 scope.go:117] "RemoveContainer" containerID="e2e854643f72ec185ece7678b0a9cf96e3fb59be63ad2e786e076a59f2c8165b"
Jul 09 18:31:07.914170 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.914131    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"]
Jul 09 18:31:07.916497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:07.916466    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs"]
Jul 09 18:31:08.896037 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:08.896001    2578 generic.go:356] "Generic (PLEG): container finished" podID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerID="d4ae11d56810243b8042f84dcfb7cfe4219bb8872bbe6be5e3a831d7d1a13ad2" exitCode=0
Jul 09 18:31:08.896432 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:08.896049    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" event={"ID":"ddbca4af-b0aa-4dd6-8cff-64b352bf184b","Type":"ContainerDied","Data":"d4ae11d56810243b8042f84dcfb7cfe4219bb8872bbe6be5e3a831d7d1a13ad2"}
Jul 09 18:31:09.404869 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:09.404822    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" path="/var/lib/kubelet/pods/efc51b02-e94c-4377-bb76-17410eb5858b/volumes"
Jul 09 18:31:09.903742 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:09.903703    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" event={"ID":"ddbca4af-b0aa-4dd6-8cff-64b352bf184b","Type":"ContainerStarted","Data":"2a9c67b54f1964e709c117fcc23148bb4dc77fa30cf07287417d02858b11e651"}
Jul 09 18:31:09.929228 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:09.929169    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" podStartSLOduration=2.806032626 podStartE2EDuration="5.929151764s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.123119138s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:31:09.927858201 +0000 UTC m=+1593.114835968" watchObservedRunningTime="2026-07-09 18:31:09.929151764 +0000 UTC m=+1593.116129558"
Jul 09 18:31:10.613482 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:10.613450    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:11.558669 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:11.558632    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:31:11.655807 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:11.655772    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:31:11.775877 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:11.775838    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:31:11.877561 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:11.877472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:31:14.262130 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:14.262093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:31:14.362335 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:14.362304    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:31:14.614037 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:14.613941    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:31:14.615587 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:14.615557    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 09 18:31:14.873304 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:14.873211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:31:24.614419 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:24.614378    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 09 18:31:34.614245 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:34.614138    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 09 18:31:36.579276 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.579238    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"]
Jul 09 18:31:36.579823 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.579702    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" containerName="main"
Jul 09 18:31:36.579823 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.579719    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" containerName="main"
Jul 09 18:31:36.579823 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.579735    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" containerName="storage-initializer"
Jul 09 18:31:36.579823 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.579744    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" containerName="storage-initializer"
Jul 09 18:31:36.580064 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.579848    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc51b02-e94c-4377-bb76-17410eb5858b" containerName="main"
Jul 09 18:31:36.582586 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.582564    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.585208 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.585186    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 09 18:31:36.596154 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.596122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"]
Jul 09 18:31:36.633866 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.633832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a536c660-613b-4de7-9415-ccdf91ae856f-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.634058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.633906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a536c660-613b-4de7-9415-ccdf91ae856f-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.634058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.633942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsw26\" (UniqueName: \"kubernetes.io/projected/a536c660-613b-4de7-9415-ccdf91ae856f-kube-api-access-fsw26\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.634058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.634014    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a536c660-613b-4de7-9415-ccdf91ae856f-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.634058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.634047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a536c660-613b-4de7-9415-ccdf91ae856f-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.634271 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.634168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a536c660-613b-4de7-9415-ccdf91ae856f-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.735636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.735595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a536c660-613b-4de7-9415-ccdf91ae856f-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.735857 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.735642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsw26\" (UniqueName: \"kubernetes.io/projected/a536c660-613b-4de7-9415-ccdf91ae856f-kube-api-access-fsw26\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.735857 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.735691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a536c660-613b-4de7-9415-ccdf91ae856f-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.735857 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.735731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a536c660-613b-4de7-9415-ccdf91ae856f-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.736036 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.735862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a536c660-613b-4de7-9415-ccdf91ae856f-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.736036 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.735929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a536c660-613b-4de7-9415-ccdf91ae856f-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.736205 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.736183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a536c660-613b-4de7-9415-ccdf91ae856f-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.736294 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.736275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a536c660-613b-4de7-9415-ccdf91ae856f-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.736366 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.736342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a536c660-613b-4de7-9415-ccdf91ae856f-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.738155 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.738127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a536c660-613b-4de7-9415-ccdf91ae856f-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.738253 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.738151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a536c660-613b-4de7-9415-ccdf91ae856f-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.748694 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.748651    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsw26\" (UniqueName: \"kubernetes.io/projected/a536c660-613b-4de7-9415-ccdf91ae856f-kube-api-access-fsw26\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" (UID: \"a536c660-613b-4de7-9415-ccdf91ae856f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:31:36.753840 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:36.753776    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:31:38.216273 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:38.216214    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:31:44.614154 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:44.614109    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 09 18:31:54.613723 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:54.613659    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 09 18:31:55.443731 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:31:55.443686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:32:03.156038 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:32:03.156000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:32:04.614233 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:32:04.614185    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 09 18:32:14.614186 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:32:14.614145    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 09 18:32:24.613893 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:32:24.613844    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 09 18:32:34.613898 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:32:34.613843    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 09 18:32:44.614343 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:32:44.614292    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 09 18:32:54.614001 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:32:54.613955    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 09 18:33:03.471813 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:33:03.471714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:33:04.614419 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:33:04.614368    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 09 18:33:14.624399 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:33:14.624360    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:33:14.632502 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:33:14.632470    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:33:39.593927 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:33:39.593882    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 18:33:39.608024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:33:39.607992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:33:45.617111 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:33:45.617077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:33:45.717664 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:33:45.717632    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:33:49.587034 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:33:49.586997    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"]
Jul 09 18:33:49.617993 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:33:49.617964    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:33:49.718151 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:33:49.718114    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:34:04.610524 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:04.610457    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main" containerID="cri-o://2a9c67b54f1964e709c117fcc23148bb4dc77fa30cf07287417d02858b11e651" gracePeriod=45
Jul 09 18:34:05.368047 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.368015    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:34:05.426654 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.426565    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-kserve-provision-location\") pod \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") "
Jul 09 18:34:05.426654 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.426605    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-model-cache\") pod \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") "
Jul 09 18:34:05.426654 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.426644    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-dshm\") pod \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") "
Jul 09 18:34:05.426954 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.426666    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftbrf\" (UniqueName: \"kubernetes.io/projected/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-kube-api-access-ftbrf\") pod \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") "
Jul 09 18:34:05.426954 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.426720    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-home\") pod \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") "
Jul 09 18:34:05.426954 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.426745    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-tmp-dir\") pod \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") "
Jul 09 18:34:05.426954 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.426810    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-tls-certs\") pod \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\" (UID: \"ddbca4af-b0aa-4dd6-8cff-64b352bf184b\") "
Jul 09 18:34:05.427166 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.426977    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ddbca4af-b0aa-4dd6-8cff-64b352bf184b" (UID: "ddbca4af-b0aa-4dd6-8cff-64b352bf184b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:05.427166 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.427090    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:05.427758 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.427726    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-home" (OuterVolumeSpecName: "home") pod "ddbca4af-b0aa-4dd6-8cff-64b352bf184b" (UID: "ddbca4af-b0aa-4dd6-8cff-64b352bf184b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:05.429050 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.429019    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-kube-api-access-ftbrf" (OuterVolumeSpecName: "kube-api-access-ftbrf") pod "ddbca4af-b0aa-4dd6-8cff-64b352bf184b" (UID: "ddbca4af-b0aa-4dd6-8cff-64b352bf184b"). InnerVolumeSpecName "kube-api-access-ftbrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:34:05.429144 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.429118    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddbca4af-b0aa-4dd6-8cff-64b352bf184b" (UID: "ddbca4af-b0aa-4dd6-8cff-64b352bf184b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:34:05.429227 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.429211    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-dshm" (OuterVolumeSpecName: "dshm") pod "ddbca4af-b0aa-4dd6-8cff-64b352bf184b" (UID: "ddbca4af-b0aa-4dd6-8cff-64b352bf184b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:05.439808 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.439771    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ddbca4af-b0aa-4dd6-8cff-64b352bf184b" (UID: "ddbca4af-b0aa-4dd6-8cff-64b352bf184b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:05.500709 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.500663    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ddbca4af-b0aa-4dd6-8cff-64b352bf184b" (UID: "ddbca4af-b0aa-4dd6-8cff-64b352bf184b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:05.528002 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.527967    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:05.528002 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.528003    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:05.528002 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.528017    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:05.528243 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.528033    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:05.528243 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.528046    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:05.528243 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.528061    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftbrf\" (UniqueName: \"kubernetes.io/projected/ddbca4af-b0aa-4dd6-8cff-64b352bf184b-kube-api-access-ftbrf\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:05.721355 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.721320    2578 generic.go:356] "Generic (PLEG): container finished" podID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerID="2a9c67b54f1964e709c117fcc23148bb4dc77fa30cf07287417d02858b11e651" exitCode=0
Jul 09 18:34:05.721748 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.721387    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"
Jul 09 18:34:05.721748 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.721399    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" event={"ID":"ddbca4af-b0aa-4dd6-8cff-64b352bf184b","Type":"ContainerDied","Data":"2a9c67b54f1964e709c117fcc23148bb4dc77fa30cf07287417d02858b11e651"}
Jul 09 18:34:05.721748 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.721464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw" event={"ID":"ddbca4af-b0aa-4dd6-8cff-64b352bf184b","Type":"ContainerDied","Data":"250326b4f6272cd02f6a9f10a132713da0b3b3dff269a4f7c363703f39bd0129"}
Jul 09 18:34:05.721748 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.721495    2578 scope.go:117] "RemoveContainer" containerID="2a9c67b54f1964e709c117fcc23148bb4dc77fa30cf07287417d02858b11e651"
Jul 09 18:34:05.730684 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.730664    2578 scope.go:117] "RemoveContainer" containerID="d4ae11d56810243b8042f84dcfb7cfe4219bb8872bbe6be5e3a831d7d1a13ad2"
Jul 09 18:34:05.739198 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.739176    2578 scope.go:117] "RemoveContainer" containerID="2a9c67b54f1964e709c117fcc23148bb4dc77fa30cf07287417d02858b11e651"
Jul 09 18:34:05.739452 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:34:05.739435    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a9c67b54f1964e709c117fcc23148bb4dc77fa30cf07287417d02858b11e651\": container with ID starting with 2a9c67b54f1964e709c117fcc23148bb4dc77fa30cf07287417d02858b11e651 not found: ID does not exist" containerID="2a9c67b54f1964e709c117fcc23148bb4dc77fa30cf07287417d02858b11e651"
Jul 09 18:34:05.739506 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.739462    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9c67b54f1964e709c117fcc23148bb4dc77fa30cf07287417d02858b11e651"} err="failed to get container status \"2a9c67b54f1964e709c117fcc23148bb4dc77fa30cf07287417d02858b11e651\": rpc error: code = NotFound desc = could not find container \"2a9c67b54f1964e709c117fcc23148bb4dc77fa30cf07287417d02858b11e651\": container with ID starting with 2a9c67b54f1964e709c117fcc23148bb4dc77fa30cf07287417d02858b11e651 not found: ID does not exist"
Jul 09 18:34:05.739506 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.739482    2578 scope.go:117] "RemoveContainer" containerID="d4ae11d56810243b8042f84dcfb7cfe4219bb8872bbe6be5e3a831d7d1a13ad2"
Jul 09 18:34:05.739712 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:34:05.739692    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ae11d56810243b8042f84dcfb7cfe4219bb8872bbe6be5e3a831d7d1a13ad2\": container with ID starting with d4ae11d56810243b8042f84dcfb7cfe4219bb8872bbe6be5e3a831d7d1a13ad2 not found: ID does not exist" containerID="d4ae11d56810243b8042f84dcfb7cfe4219bb8872bbe6be5e3a831d7d1a13ad2"
Jul 09 18:34:05.739768 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.739723    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ae11d56810243b8042f84dcfb7cfe4219bb8872bbe6be5e3a831d7d1a13ad2"} err="failed to get container status \"d4ae11d56810243b8042f84dcfb7cfe4219bb8872bbe6be5e3a831d7d1a13ad2\": rpc error: code = NotFound desc = could not find container \"d4ae11d56810243b8042f84dcfb7cfe4219bb8872bbe6be5e3a831d7d1a13ad2\": container with ID starting with d4ae11d56810243b8042f84dcfb7cfe4219bb8872bbe6be5e3a831d7d1a13ad2 not found: ID does not exist"
Jul 09 18:34:05.746201 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.746178    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"]
Jul 09 18:34:05.750253 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:05.750219    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw"]
Jul 09 18:34:07.403612 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:07.403577    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" path="/var/lib/kubelet/pods/ddbca4af-b0aa-4dd6-8cff-64b352bf184b/volumes"
Jul 09 18:34:12.667623 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:12.667584    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"]
Jul 09 18:34:12.668209 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:12.667780    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs" podUID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" containerName="istio-proxy" containerID="cri-o://8b89b1462854051443c9337dc19247a21c69730666ca6d4878788fcd5e26ec4d" gracePeriod=30
Jul 09 18:34:17.924880 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:17.924850    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:34:18.042086 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.041981    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrmsj\" (UniqueName: \"kubernetes.io/projected/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-kube-api-access-rrmsj\") pod \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") "
Jul 09 18:34:18.042086 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.042021    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-workload-certs\") pod \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") "
Jul 09 18:34:18.042086 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.042044    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-podinfo\") pod \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") "
Jul 09 18:34:18.042388 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.042104    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-envoy\") pod \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") "
Jul 09 18:34:18.042388 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.042127    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-token\") pod \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") "
Jul 09 18:34:18.042388 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.042156    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-data\") pod \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") "
Jul 09 18:34:18.042388 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.042182    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istiod-ca-cert\") pod \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") "
Jul 09 18:34:18.042388 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.042220    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-workload-socket\") pod \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") "
Jul 09 18:34:18.042388 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.042282    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-credential-socket\") pod \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\" (UID: \"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d\") "
Jul 09 18:34:18.042388 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.042376    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" (UID: "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:18.042750 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.042602    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" (UID: "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:18.042750 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.042606    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-workload-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:18.042901 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.042827    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" (UID: "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:18.042901 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.042827    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" (UID: "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:18.042976 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.042934    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" (UID: "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:34:18.044407 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.044379    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-kube-api-access-rrmsj" (OuterVolumeSpecName: "kube-api-access-rrmsj") pod "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" (UID: "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d"). InnerVolumeSpecName "kube-api-access-rrmsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:34:18.044556 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.044534    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" (UID: "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:18.044772 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.044745    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" (UID: "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:34:18.044874 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.044849    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" (UID: "3ddb135e-1993-48c8-9b70-ed8f86d6bc8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 18:34:18.144058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.144018    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrmsj\" (UniqueName: \"kubernetes.io/projected/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-kube-api-access-rrmsj\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:18.144058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.144047    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-podinfo\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:18.144058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.144058    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-envoy\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:18.144058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.144066    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-token\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:18.144332 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.144074    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istio-data\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:18.144332 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.144082    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-istiod-ca-cert\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:18.144332 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.144089    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-workload-socket\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:18.144332 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.144097    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d-credential-socket\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:34:18.784893 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.784860    2578 generic.go:356] "Generic (PLEG): container finished" podID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" containerID="8b89b1462854051443c9337dc19247a21c69730666ca6d4878788fcd5e26ec4d" exitCode=0
Jul 09 18:34:18.785070 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.784905    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"
Jul 09 18:34:18.785070 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.784940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs" event={"ID":"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d","Type":"ContainerDied","Data":"8b89b1462854051443c9337dc19247a21c69730666ca6d4878788fcd5e26ec4d"}
Jul 09 18:34:18.785070 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.784990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs" event={"ID":"3ddb135e-1993-48c8-9b70-ed8f86d6bc8d","Type":"ContainerDied","Data":"cd36b5d06808791d0f1e94337360a6533769826af841cff2fdc64ff821480457"}
Jul 09 18:34:18.785070 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.785013    2578 scope.go:117] "RemoveContainer" containerID="8b89b1462854051443c9337dc19247a21c69730666ca6d4878788fcd5e26ec4d"
Jul 09 18:34:18.794011 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.793993    2578 scope.go:117] "RemoveContainer" containerID="e03ddd03cf6f20377a900c84467f0dcffa72bd772b8d6c968c7587e17584330e"
Jul 09 18:34:18.802001 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.801980    2578 scope.go:117] "RemoveContainer" containerID="8b89b1462854051443c9337dc19247a21c69730666ca6d4878788fcd5e26ec4d"
Jul 09 18:34:18.802303 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:34:18.802283    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b89b1462854051443c9337dc19247a21c69730666ca6d4878788fcd5e26ec4d\": container with ID starting with 8b89b1462854051443c9337dc19247a21c69730666ca6d4878788fcd5e26ec4d not found: ID does not exist" containerID="8b89b1462854051443c9337dc19247a21c69730666ca6d4878788fcd5e26ec4d"
Jul 09 18:34:18.802350 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.802317    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b89b1462854051443c9337dc19247a21c69730666ca6d4878788fcd5e26ec4d"} err="failed to get container status \"8b89b1462854051443c9337dc19247a21c69730666ca6d4878788fcd5e26ec4d\": rpc error: code = NotFound desc = could not find container \"8b89b1462854051443c9337dc19247a21c69730666ca6d4878788fcd5e26ec4d\": container with ID starting with 8b89b1462854051443c9337dc19247a21c69730666ca6d4878788fcd5e26ec4d not found: ID does not exist"
Jul 09 18:34:18.802350 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.802337    2578 scope.go:117] "RemoveContainer" containerID="e03ddd03cf6f20377a900c84467f0dcffa72bd772b8d6c968c7587e17584330e"
Jul 09 18:34:18.802610 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:34:18.802592    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03ddd03cf6f20377a900c84467f0dcffa72bd772b8d6c968c7587e17584330e\": container with ID starting with e03ddd03cf6f20377a900c84467f0dcffa72bd772b8d6c968c7587e17584330e not found: ID does not exist" containerID="e03ddd03cf6f20377a900c84467f0dcffa72bd772b8d6c968c7587e17584330e"
Jul 09 18:34:18.802669 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.802616    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03ddd03cf6f20377a900c84467f0dcffa72bd772b8d6c968c7587e17584330e"} err="failed to get container status \"e03ddd03cf6f20377a900c84467f0dcffa72bd772b8d6c968c7587e17584330e\": rpc error: code = NotFound desc = could not find container \"e03ddd03cf6f20377a900c84467f0dcffa72bd772b8d6c968c7587e17584330e\": container with ID starting with e03ddd03cf6f20377a900c84467f0dcffa72bd772b8d6c968c7587e17584330e not found: ID does not exist"
Jul 09 18:34:18.807899 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.807875    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"]
Jul 09 18:34:18.813640 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:18.813616    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs"]
Jul 09 18:34:19.403220 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:19.403184    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" path="/var/lib/kubelet/pods/3ddb135e-1993-48c8-9b70-ed8f86d6bc8d/volumes"
Jul 09 18:34:24.098863 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.098745    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"]
Jul 09 18:34:24.099225 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.099067    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="storage-initializer"
Jul 09 18:34:24.099225 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.099079    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="storage-initializer"
Jul 09 18:34:24.099225 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.099091    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" containerName="istio-proxy"
Jul 09 18:34:24.099225 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.099097    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" containerName="istio-proxy"
Jul 09 18:34:24.099225 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.099105    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main"
Jul 09 18:34:24.099225 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.099111    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main"
Jul 09 18:34:24.099225 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.099188    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" containerName="istio-proxy"
Jul 09 18:34:24.099225 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.099196    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" containerName="istio-proxy"
Jul 09 18:34:24.099225 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.099203    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddbca4af-b0aa-4dd6-8cff-64b352bf184b" containerName="main"
Jul 09 18:34:24.099500 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.099262    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" containerName="istio-proxy"
Jul 09 18:34:24.099500 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.099268    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ddb135e-1993-48c8-9b70-ed8f86d6bc8d" containerName="istio-proxy"
Jul 09 18:34:24.103334 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.103312    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.108734 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.108710    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:34:24.110305 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.110286    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 09 18:34:24.119018 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.118988    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"]
Jul 09 18:34:24.186166 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.186130    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:34:24.203333 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.203295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fwbt\" (UniqueName: \"kubernetes.io/projected/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-kube-api-access-9fwbt\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.203488 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.203365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-tmp-dir\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.203488 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.203446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-kserve-provision-location\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.203630 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.203493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-model-cache\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.203630 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.203549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-home\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.203630 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.203573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-dshm\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.203630 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.203620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-tls-certs\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.304891 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.304852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-home\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.304891 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.304889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-dshm\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.305152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.304917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-tls-certs\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.305152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.304940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fwbt\" (UniqueName: \"kubernetes.io/projected/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-kube-api-access-9fwbt\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.305152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.305003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-tmp-dir\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.305152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.305029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-kserve-provision-location\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.305152 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.305054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-model-cache\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.305398 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.305322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-home\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.305398 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.305383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-tmp-dir\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.305515 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.305495    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-model-cache\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.305555 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.305513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-kserve-provision-location\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.307384 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.307359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-dshm\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.307595 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.307580    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-tls-certs\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.314235 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.314208    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fwbt\" (UniqueName: \"kubernetes.io/projected/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-kube-api-access-9fwbt\") pod \"router-with-refs-test-kserve-7777648d4d-6lsph\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.414022 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.413919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:24.563664 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.563628    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"]
Jul 09 18:34:24.566902 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:34:24.566869    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5698e7ad_57b2_441e_9fbf_ef1d0add8cb9.slice/crio-38bc3b4b7bd5083d0c9de4bb155b1a4843a3f6b54d15817783901d57869a35ad WatchSource:0}: Error finding container 38bc3b4b7bd5083d0c9de4bb155b1a4843a3f6b54d15817783901d57869a35ad: Status 404 returned error can't find the container with id 38bc3b4b7bd5083d0c9de4bb155b1a4843a3f6b54d15817783901d57869a35ad
Jul 09 18:34:24.812678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.812635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" event={"ID":"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9","Type":"ContainerStarted","Data":"c634c446e2ff0fe498216334a2554a75877c729f102f490b167d6996541459b2"}
Jul 09 18:34:24.812873 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.812700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" event={"ID":"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9","Type":"ContainerStarted","Data":"38bc3b4b7bd5083d0c9de4bb155b1a4843a3f6b54d15817783901d57869a35ad"}
Jul 09 18:34:24.883568 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.883527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:34:24.963773 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:24.963738    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:34:28.838459 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:28.838420    2578 generic.go:356] "Generic (PLEG): container finished" podID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerID="c634c446e2ff0fe498216334a2554a75877c729f102f490b167d6996541459b2" exitCode=0
Jul 09 18:34:28.838854 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:28.838496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" event={"ID":"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9","Type":"ContainerDied","Data":"c634c446e2ff0fe498216334a2554a75877c729f102f490b167d6996541459b2"}
Jul 09 18:34:29.845482 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:29.845447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" event={"ID":"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9","Type":"ContainerStarted","Data":"3c9e5758bb6f3a38d51aeb9cf394f8f8c202447a8927e1187df77d7917f6b728"}
Jul 09 18:34:29.875849 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:29.875774    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podStartSLOduration=2.221506306 podStartE2EDuration="5.875756908s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.654250602s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:34:29.873341395 +0000 UTC m=+1793.060319183" watchObservedRunningTime="2026-07-09 18:34:29.875756908 +0000 UTC m=+1793.062734676"
Jul 09 18:34:30.414527 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:30.414486    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:34.414745 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:34.414700    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:34:34.416032 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:34.416003    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 09 18:34:37.377481 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:37.377444    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:34:37.383883 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:37.383848    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:34:37.387171 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:37.387132    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:34:37.392918 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:37.392894    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:34:44.415011 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:44.414960    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 09 18:34:54.414733 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:34:54.414676    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 09 18:35:04.415496 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:35:04.415433    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 09 18:35:14.415455 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:35:14.415412    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 09 18:35:24.415495 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:35:24.415437    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 09 18:35:34.414390 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:35:34.414338    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 09 18:35:42.608760 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:35:42.608710    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 18:35:43.174712 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:35:43.174669    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:35:44.414458 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:35:44.414409    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 09 18:35:54.415166 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:35:54.415122    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 09 18:36:04.415039 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:36:04.414940    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 09 18:36:14.414431 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:36:14.414387    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 09 18:36:24.414383 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:36:24.414334    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 09 18:36:34.425249 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:36:34.425217    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:36:34.433244 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:36:34.433214    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:36:47.024745 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:36:47.024698    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"]
Jul 09 18:36:47.155782 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:36:47.155739    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:36:48.998149 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:36:48.998059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:37:02.047935 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.047868    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main" containerID="cri-o://3c9e5758bb6f3a38d51aeb9cf394f8f8c202447a8927e1187df77d7917f6b728" gracePeriod=45
Jul 09 18:37:02.426199 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.426108    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 09 18:37:02.800188 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.800154    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:37:02.926682 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.926645    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-tls-certs\") pod \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") "
Jul 09 18:37:02.926897 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.926705    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-tmp-dir\") pod \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") "
Jul 09 18:37:02.926897 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.926731    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-model-cache\") pod \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") "
Jul 09 18:37:02.926897 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.926751    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-home\") pod \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") "
Jul 09 18:37:02.926897 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.926859    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-dshm\") pod \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") "
Jul 09 18:37:02.927084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.926900    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-kserve-provision-location\") pod \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") "
Jul 09 18:37:02.927084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.926956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fwbt\" (UniqueName: \"kubernetes.io/projected/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-kube-api-access-9fwbt\") pod \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\" (UID: \"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9\") "
Jul 09 18:37:02.927084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.927054    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-model-cache" (OuterVolumeSpecName: "model-cache") pod "5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" (UID: "5698e7ad-57b2-441e-9fbf-ef1d0add8cb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:37:02.927332 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.927311    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:37:02.927553 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.927527    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-home" (OuterVolumeSpecName: "home") pod "5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" (UID: "5698e7ad-57b2-441e-9fbf-ef1d0add8cb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:37:02.928949 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.928928    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-dshm" (OuterVolumeSpecName: "dshm") pod "5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" (UID: "5698e7ad-57b2-441e-9fbf-ef1d0add8cb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:37:02.929414 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.929392    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" (UID: "5698e7ad-57b2-441e-9fbf-ef1d0add8cb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:37:02.929414 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.929404    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-kube-api-access-9fwbt" (OuterVolumeSpecName: "kube-api-access-9fwbt") pod "5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" (UID: "5698e7ad-57b2-441e-9fbf-ef1d0add8cb9"). InnerVolumeSpecName "kube-api-access-9fwbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:37:02.939491 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.939459    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" (UID: "5698e7ad-57b2-441e-9fbf-ef1d0add8cb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:37:02.995780 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:02.995670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" (UID: "5698e7ad-57b2-441e-9fbf-ef1d0add8cb9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:37:03.027941 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.027908    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:37:03.027941 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.027938    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:37:03.028084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.027947    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:37:03.028084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.027956    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:37:03.028084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.027966    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fwbt\" (UniqueName: \"kubernetes.io/projected/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-kube-api-access-9fwbt\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:37:03.028084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.027974    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:37:03.525371 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.525336    2578 generic.go:356] "Generic (PLEG): container finished" podID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerID="3c9e5758bb6f3a38d51aeb9cf394f8f8c202447a8927e1187df77d7917f6b728" exitCode=0
Jul 09 18:37:03.525961 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.525391    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"
Jul 09 18:37:03.525961 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.525428    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" event={"ID":"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9","Type":"ContainerDied","Data":"3c9e5758bb6f3a38d51aeb9cf394f8f8c202447a8927e1187df77d7917f6b728"}
Jul 09 18:37:03.525961 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.525499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph" event={"ID":"5698e7ad-57b2-441e-9fbf-ef1d0add8cb9","Type":"ContainerDied","Data":"38bc3b4b7bd5083d0c9de4bb155b1a4843a3f6b54d15817783901d57869a35ad"}
Jul 09 18:37:03.525961 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.525536    2578 scope.go:117] "RemoveContainer" containerID="3c9e5758bb6f3a38d51aeb9cf394f8f8c202447a8927e1187df77d7917f6b728"
Jul 09 18:37:03.534194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.534164    2578 scope.go:117] "RemoveContainer" containerID="c634c446e2ff0fe498216334a2554a75877c729f102f490b167d6996541459b2"
Jul 09 18:37:03.542776 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.542744    2578 scope.go:117] "RemoveContainer" containerID="3c9e5758bb6f3a38d51aeb9cf394f8f8c202447a8927e1187df77d7917f6b728"
Jul 09 18:37:03.543144 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:37:03.543124    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c9e5758bb6f3a38d51aeb9cf394f8f8c202447a8927e1187df77d7917f6b728\": container with ID starting with 3c9e5758bb6f3a38d51aeb9cf394f8f8c202447a8927e1187df77d7917f6b728 not found: ID does not exist" containerID="3c9e5758bb6f3a38d51aeb9cf394f8f8c202447a8927e1187df77d7917f6b728"
Jul 09 18:37:03.543207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.543155    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9e5758bb6f3a38d51aeb9cf394f8f8c202447a8927e1187df77d7917f6b728"} err="failed to get container status \"3c9e5758bb6f3a38d51aeb9cf394f8f8c202447a8927e1187df77d7917f6b728\": rpc error: code = NotFound desc = could not find container \"3c9e5758bb6f3a38d51aeb9cf394f8f8c202447a8927e1187df77d7917f6b728\": container with ID starting with 3c9e5758bb6f3a38d51aeb9cf394f8f8c202447a8927e1187df77d7917f6b728 not found: ID does not exist"
Jul 09 18:37:03.543207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.543176    2578 scope.go:117] "RemoveContainer" containerID="c634c446e2ff0fe498216334a2554a75877c729f102f490b167d6996541459b2"
Jul 09 18:37:03.543457 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:37:03.543441    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c634c446e2ff0fe498216334a2554a75877c729f102f490b167d6996541459b2\": container with ID starting with c634c446e2ff0fe498216334a2554a75877c729f102f490b167d6996541459b2 not found: ID does not exist" containerID="c634c446e2ff0fe498216334a2554a75877c729f102f490b167d6996541459b2"
Jul 09 18:37:03.543500 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.543461    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c634c446e2ff0fe498216334a2554a75877c729f102f490b167d6996541459b2"} err="failed to get container status \"c634c446e2ff0fe498216334a2554a75877c729f102f490b167d6996541459b2\": rpc error: code = NotFound desc = could not find container \"c634c446e2ff0fe498216334a2554a75877c729f102f490b167d6996541459b2\": container with ID starting with c634c446e2ff0fe498216334a2554a75877c729f102f490b167d6996541459b2 not found: ID does not exist"
Jul 09 18:37:03.546502 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.546470    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"]
Jul 09 18:37:03.550038 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:03.550011    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph"]
Jul 09 18:37:05.403285 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:05.403251    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" path="/var/lib/kubelet/pods/5698e7ad-57b2-441e-9fbf-ef1d0add8cb9/volumes"
Jul 09 18:37:15.863414 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.863376    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"]
Jul 09 18:37:15.863987 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.863685    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="storage-initializer"
Jul 09 18:37:15.863987 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.863696    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="storage-initializer"
Jul 09 18:37:15.863987 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.863710    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main"
Jul 09 18:37:15.863987 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.863715    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main"
Jul 09 18:37:15.863987 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.863775    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5698e7ad-57b2-441e-9fbf-ef1d0add8cb9" containerName="main"
Jul 09 18:37:15.866816 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.866769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:15.870049 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.870023    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:37:15.870049 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.870045    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-z4k7h\""
Jul 09 18:37:15.870263 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.870047    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 09 18:37:15.881384 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.881347    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"]
Jul 09 18:37:15.884697 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.884672    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:15.894317 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.894291    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"]
Jul 09 18:37:15.907103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.907069    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"]
Jul 09 18:37:15.919552 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.919513    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:37:15.930920 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.930885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cgsg\" (UniqueName: \"kubernetes.io/projected/e6111935-17ca-4c68-a361-740869af41e5-kube-api-access-7cgsg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:15.931103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.930977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:15.931103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.930999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:15.931103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.931023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:15.931103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.931055    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:15.931103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.931082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2fzp\" (UniqueName: \"kubernetes.io/projected/229e4f3b-dac2-4182-bb2d-96fef9573535-kube-api-access-f2fzp\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:15.931364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.931155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:15.931364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.931196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:15.931364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.931228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:15.931364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.931264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:15.931364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.931310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6111935-17ca-4c68-a361-740869af41e5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:15.931364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.931330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/229e4f3b-dac2-4182-bb2d-96fef9573535-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:15.931364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.931352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:15.931599 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:15.931384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.032560 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.032521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.032560 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.032563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6111935-17ca-4c68-a361-740869af41e5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.032560 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.032580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/229e4f3b-dac2-4182-bb2d-96fef9573535-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.032948 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.032601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.032948 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.032655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.032948 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.032702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cgsg\" (UniqueName: \"kubernetes.io/projected/e6111935-17ca-4c68-a361-740869af41e5-kube-api-access-7cgsg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.032948 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.032786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.032948 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.032853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.032948 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.032880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.032948 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.032931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.033304 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.032972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2fzp\" (UniqueName: \"kubernetes.io/projected/229e4f3b-dac2-4182-bb2d-96fef9573535-kube-api-access-f2fzp\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.033304 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.033018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.033304 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.033050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.033304 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.033070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.033304 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.033081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.033304 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.033089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.033304 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.033150    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.033304 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.033071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.033714 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.033307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.033714 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.033464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.033714 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.033479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.033714 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.033501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.035396 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.035371    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.035534 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.035416    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.035534 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.035486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6111935-17ca-4c68-a361-740869af41e5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.035755 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.035736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/229e4f3b-dac2-4182-bb2d-96fef9573535-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.043316 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.043276    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2fzp\" (UniqueName: \"kubernetes.io/projected/229e4f3b-dac2-4182-bb2d-96fef9573535-kube-api-access-f2fzp\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.045964 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.045932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cgsg\" (UniqueName: \"kubernetes.io/projected/e6111935-17ca-4c68-a361-740869af41e5-kube-api-access-7cgsg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.176376 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.176282    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:16.196630 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.196591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:16.341032 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.341001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"]
Jul 09 18:37:16.343649 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:37:16.343614    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6111935_17ca_4c68_a361_740869af41e5.slice/crio-1b35d974cd2ef19f938c638b43229fbc910f744d094bdb0e98e77f38a7bf7ed7 WatchSource:0}: Error finding container 1b35d974cd2ef19f938c638b43229fbc910f744d094bdb0e98e77f38a7bf7ed7: Status 404 returned error can't find the container with id 1b35d974cd2ef19f938c638b43229fbc910f744d094bdb0e98e77f38a7bf7ed7
Jul 09 18:37:16.345526 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.345511    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:37:16.363214 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.363178    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"]
Jul 09 18:37:16.366943 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:37:16.366916    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229e4f3b_dac2_4182_bb2d_96fef9573535.slice/crio-073f0d971179bcb1af2d31b44443938cb44ebcec64a4122c37788302cd97224a WatchSource:0}: Error finding container 073f0d971179bcb1af2d31b44443938cb44ebcec64a4122c37788302cd97224a: Status 404 returned error can't find the container with id 073f0d971179bcb1af2d31b44443938cb44ebcec64a4122c37788302cd97224a
Jul 09 18:37:16.375391 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.375365    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:37:16.583299 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.583259    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" event={"ID":"229e4f3b-dac2-4182-bb2d-96fef9573535","Type":"ContainerStarted","Data":"4984dffc8cb64e109aa321f0f98cd821be42fa0b9d1b753731c90fb105297e00"}
Jul 09 18:37:16.583495 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.583376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" event={"ID":"229e4f3b-dac2-4182-bb2d-96fef9573535","Type":"ContainerStarted","Data":"073f0d971179bcb1af2d31b44443938cb44ebcec64a4122c37788302cd97224a"}
Jul 09 18:37:16.585023 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:16.584992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" event={"ID":"e6111935-17ca-4c68-a361-740869af41e5","Type":"ContainerStarted","Data":"1b35d974cd2ef19f938c638b43229fbc910f744d094bdb0e98e77f38a7bf7ed7"}
Jul 09 18:37:17.591029 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:17.590991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" event={"ID":"e6111935-17ca-4c68-a361-740869af41e5","Type":"ContainerStarted","Data":"3af432e30338188e6582755d4aaa4e5f332c338f9b30b536a80410d4d5a1a18a"}
Jul 09 18:37:17.591504 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:17.591233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:18.597951 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:18.597913    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" event={"ID":"e6111935-17ca-4c68-a361-740869af41e5","Type":"ContainerStarted","Data":"e5fc516eac94018bdceccc6b7e6af4c110fa1c2cab31815ab6c378a509258d69"}
Jul 09 18:37:22.618673 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:22.618631    2578 generic.go:356] "Generic (PLEG): container finished" podID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerID="4984dffc8cb64e109aa321f0f98cd821be42fa0b9d1b753731c90fb105297e00" exitCode=0
Jul 09 18:37:22.619255 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:22.618741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" event={"ID":"229e4f3b-dac2-4182-bb2d-96fef9573535","Type":"ContainerDied","Data":"4984dffc8cb64e109aa321f0f98cd821be42fa0b9d1b753731c90fb105297e00"}
Jul 09 18:37:23.625194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:23.625155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" event={"ID":"229e4f3b-dac2-4182-bb2d-96fef9573535","Type":"ContainerStarted","Data":"70e9590e680e146c2c00fe4624b6423f739677a5cb12dd98b6083be69c2f6aa8"}
Jul 09 18:37:23.627142 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:23.627116    2578 generic.go:356] "Generic (PLEG): container finished" podID="e6111935-17ca-4c68-a361-740869af41e5" containerID="e5fc516eac94018bdceccc6b7e6af4c110fa1c2cab31815ab6c378a509258d69" exitCode=0
Jul 09 18:37:23.627278 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:23.627180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" event={"ID":"e6111935-17ca-4c68-a361-740869af41e5","Type":"ContainerDied","Data":"e5fc516eac94018bdceccc6b7e6af4c110fa1c2cab31815ab6c378a509258d69"}
Jul 09 18:37:23.655294 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:23.655184    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podStartSLOduration=3.229940305 podStartE2EDuration="8.655160369s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.425220064s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:37:23.652173161 +0000 UTC m=+1966.839150929" watchObservedRunningTime="2026-07-09 18:37:23.655160369 +0000 UTC m=+1966.842138139"
Jul 09 18:37:24.634845 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:24.634787    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" event={"ID":"e6111935-17ca-4c68-a361-740869af41e5","Type":"ContainerStarted","Data":"08785b20c6b65b00f472978fcd5f6f6f7e24eb8cc40ec6157a663e654aaa35a1"}
Jul 09 18:37:24.660933 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:24.660870    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podStartSLOduration=3.761811356 podStartE2EDuration="9.660848085s" totalImagesPullingTime="740.060299ms" totalInitContainerRuntime="5.15897643s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:37:24.658681121 +0000 UTC m=+1967.845658889" watchObservedRunningTime="2026-07-09 18:37:24.660848085 +0000 UTC m=+1967.847825854"
Jul 09 18:37:26.177170 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:26.177117    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:26.177622 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:26.177208    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:26.178751 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:26.178713    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:37:26.196846 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:26.196786    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:26.197030 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:26.196892    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:37:26.198409 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:26.198377    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:37:36.177132 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:36.177083    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:37:36.197249 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:36.191397    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:37:36.197496 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:36.197464    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:37:37.113400 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:37.113347    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:37:46.176091 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:37:46.176032    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 18:37:46.177303 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:46.177261    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:37:46.197733 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:46.197690    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:37:46.749578 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:46.749545    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:37:56.177055 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:56.176988    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:37:56.198463 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:37:56.198422    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:38:06.177398 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:38:06.177341    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:38:06.197596 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:38:06.197544    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:38:13.894789 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:38:13.894738    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:38:16.177418 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:38:16.177372    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:38:16.197023 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:38:16.196979    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:38:26.176756 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:38:26.176702    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:38:26.198370 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:38:26.198327    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:38:36.177591 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:38:36.177545    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:38:36.197525 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:38:36.197472    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:38:46.177706 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:38:46.177658    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:38:46.197513 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:38:46.197466    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:38:56.176973 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:38:56.176917    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:38:56.197902 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:38:56.197863    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:39:06.177332 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:06.177228    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:39:06.197296 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:06.197247    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:39:16.177193 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:16.177137    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:39:16.197232 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:16.197181    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:39:26.176748 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:26.176699    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:39:26.197693 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:26.197650    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:39:36.177226 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:36.177177    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:39:36.197765 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:36.197719    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:39:37.408485 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:37.408451    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:39:37.413359 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:37.413330    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:39:37.416851 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:37.416826    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:39:37.421273 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:37.421253    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:39:46.177021 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:46.176960    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:39:46.197753 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:46.197707    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:39:49.751010 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:39:49.750965    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 18:39:50.354675 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:50.354640    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:39:53.771076 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:53.771031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:39:56.177056 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:56.177002    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:39:56.198370 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:39:56.198327    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:40:06.177174 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:06.177119    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Jul 09 18:40:06.197653 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:06.197580    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 09 18:40:16.186866 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:16.186822    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:40:16.199363 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:16.199332    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:40:16.208043 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:16.208018    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:40:16.215987 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:16.215945    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:40:31.685612 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:31.685508    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"]
Jul 09 18:40:31.698570 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:31.698529    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"]
Jul 09 18:40:31.714213 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:31.714146    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:40:31.859324 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:31.859285    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:40:46.711420 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:46.711369    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main" containerID="cri-o://08785b20c6b65b00f472978fcd5f6f6f7e24eb8cc40ec6157a663e654aaa35a1" gracePeriod=45
Jul 09 18:40:46.723482 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:46.723438    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main" containerID="cri-o://70e9590e680e146c2c00fe4624b6423f739677a5cb12dd98b6083be69c2f6aa8" gracePeriod=45
Jul 09 18:40:47.536428 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.536382    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="llm-d-routing-sidecar" containerID="cri-o://3af432e30338188e6582755d4aaa4e5f332c338f9b30b536a80410d4d5a1a18a" gracePeriod=45
Jul 09 18:40:47.636938 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.636905    2578 generic.go:356] "Generic (PLEG): container finished" podID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerID="70e9590e680e146c2c00fe4624b6423f739677a5cb12dd98b6083be69c2f6aa8" exitCode=0
Jul 09 18:40:47.637120 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.636974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" event={"ID":"229e4f3b-dac2-4182-bb2d-96fef9573535","Type":"ContainerDied","Data":"70e9590e680e146c2c00fe4624b6423f739677a5cb12dd98b6083be69c2f6aa8"}
Jul 09 18:40:47.639234 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.639199    2578 generic.go:356] "Generic (PLEG): container finished" podID="e6111935-17ca-4c68-a361-740869af41e5" containerID="08785b20c6b65b00f472978fcd5f6f6f7e24eb8cc40ec6157a663e654aaa35a1" exitCode=0
Jul 09 18:40:47.639365 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.639238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" event={"ID":"e6111935-17ca-4c68-a361-740869af41e5","Type":"ContainerDied","Data":"08785b20c6b65b00f472978fcd5f6f6f7e24eb8cc40ec6157a663e654aaa35a1"}
Jul 09 18:40:47.691854 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.691827    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:40:47.749754 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.749648    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-kserve-provision-location\") pod \"229e4f3b-dac2-4182-bb2d-96fef9573535\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") "
Jul 09 18:40:47.749754 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.749716    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-tmp-dir\") pod \"229e4f3b-dac2-4182-bb2d-96fef9573535\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") "
Jul 09 18:40:47.749754 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.749733    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/229e4f3b-dac2-4182-bb2d-96fef9573535-tls-certs\") pod \"229e4f3b-dac2-4182-bb2d-96fef9573535\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") "
Jul 09 18:40:47.750594 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.749760    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2fzp\" (UniqueName: \"kubernetes.io/projected/229e4f3b-dac2-4182-bb2d-96fef9573535-kube-api-access-f2fzp\") pod \"229e4f3b-dac2-4182-bb2d-96fef9573535\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") "
Jul 09 18:40:47.750594 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.749788    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-model-cache\") pod \"229e4f3b-dac2-4182-bb2d-96fef9573535\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") "
Jul 09 18:40:47.750594 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.749840    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-dshm\") pod \"229e4f3b-dac2-4182-bb2d-96fef9573535\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") "
Jul 09 18:40:47.750594 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.749919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-home\") pod \"229e4f3b-dac2-4182-bb2d-96fef9573535\" (UID: \"229e4f3b-dac2-4182-bb2d-96fef9573535\") "
Jul 09 18:40:47.750594 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.750476    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-model-cache" (OuterVolumeSpecName: "model-cache") pod "229e4f3b-dac2-4182-bb2d-96fef9573535" (UID: "229e4f3b-dac2-4182-bb2d-96fef9573535"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:47.750930 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.750903    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-home" (OuterVolumeSpecName: "home") pod "229e4f3b-dac2-4182-bb2d-96fef9573535" (UID: "229e4f3b-dac2-4182-bb2d-96fef9573535"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:47.752488 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.752457    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229e4f3b-dac2-4182-bb2d-96fef9573535-kube-api-access-f2fzp" (OuterVolumeSpecName: "kube-api-access-f2fzp") pod "229e4f3b-dac2-4182-bb2d-96fef9573535" (UID: "229e4f3b-dac2-4182-bb2d-96fef9573535"). InnerVolumeSpecName "kube-api-access-f2fzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:40:47.752488 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.752481    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229e4f3b-dac2-4182-bb2d-96fef9573535-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "229e4f3b-dac2-4182-bb2d-96fef9573535" (UID: "229e4f3b-dac2-4182-bb2d-96fef9573535"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:40:47.752717 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.752695    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-dshm" (OuterVolumeSpecName: "dshm") pod "229e4f3b-dac2-4182-bb2d-96fef9573535" (UID: "229e4f3b-dac2-4182-bb2d-96fef9573535"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:47.763192 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.763156    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "229e4f3b-dac2-4182-bb2d-96fef9573535" (UID: "229e4f3b-dac2-4182-bb2d-96fef9573535"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:47.807946 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.807912    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:40:47.820644 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.820607    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "229e4f3b-dac2-4182-bb2d-96fef9573535" (UID: "229e4f3b-dac2-4182-bb2d-96fef9573535"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:47.850536 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.850505    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cgsg\" (UniqueName: \"kubernetes.io/projected/e6111935-17ca-4c68-a361-740869af41e5-kube-api-access-7cgsg\") pod \"e6111935-17ca-4c68-a361-740869af41e5\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") "
Jul 09 18:40:47.850727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.850551    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-tmp-dir\") pod \"e6111935-17ca-4c68-a361-740869af41e5\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") "
Jul 09 18:40:47.850727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.850634    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-home\") pod \"e6111935-17ca-4c68-a361-740869af41e5\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") "
Jul 09 18:40:47.850727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.850670    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-dshm\") pod \"e6111935-17ca-4c68-a361-740869af41e5\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") "
Jul 09 18:40:47.850727 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.850712    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6111935-17ca-4c68-a361-740869af41e5-tls-certs\") pod \"e6111935-17ca-4c68-a361-740869af41e5\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") "
Jul 09 18:40:47.851046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.850749    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-model-cache\") pod \"e6111935-17ca-4c68-a361-740869af41e5\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") "
Jul 09 18:40:47.851046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.850842    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-kserve-provision-location\") pod \"e6111935-17ca-4c68-a361-740869af41e5\" (UID: \"e6111935-17ca-4c68-a361-740869af41e5\") "
Jul 09 18:40:47.851147 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.851123    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:40:47.851147 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.851141    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:40:47.851250 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.851155    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:40:47.851250 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.851170    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:40:47.851250 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.851183    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/229e4f3b-dac2-4182-bb2d-96fef9573535-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:40:47.851250 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.851196    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2fzp\" (UniqueName: \"kubernetes.io/projected/229e4f3b-dac2-4182-bb2d-96fef9573535-kube-api-access-f2fzp\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:40:47.851250 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.851210    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/229e4f3b-dac2-4182-bb2d-96fef9573535-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:40:47.851496 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.851456    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6111935-17ca-4c68-a361-740869af41e5" (UID: "e6111935-17ca-4c68-a361-740869af41e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:47.851654 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.851599    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-home" (OuterVolumeSpecName: "home") pod "e6111935-17ca-4c68-a361-740869af41e5" (UID: "e6111935-17ca-4c68-a361-740869af41e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:47.853321 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.853288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6111935-17ca-4c68-a361-740869af41e5-kube-api-access-7cgsg" (OuterVolumeSpecName: "kube-api-access-7cgsg") pod "e6111935-17ca-4c68-a361-740869af41e5" (UID: "e6111935-17ca-4c68-a361-740869af41e5"). InnerVolumeSpecName "kube-api-access-7cgsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:40:47.853603 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.853569    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-dshm" (OuterVolumeSpecName: "dshm") pod "e6111935-17ca-4c68-a361-740869af41e5" (UID: "e6111935-17ca-4c68-a361-740869af41e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:47.853842 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.853775    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6111935-17ca-4c68-a361-740869af41e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6111935-17ca-4c68-a361-740869af41e5" (UID: "e6111935-17ca-4c68-a361-740869af41e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:40:47.866715 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.866676    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6111935-17ca-4c68-a361-740869af41e5" (UID: "e6111935-17ca-4c68-a361-740869af41e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:47.916206 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.916158    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6111935-17ca-4c68-a361-740869af41e5" (UID: "e6111935-17ca-4c68-a361-740869af41e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:47.951939 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.951899    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:40:47.951939 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.951928    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:40:47.951939 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.951938    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:40:47.951939 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.951946    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6111935-17ca-4c68-a361-740869af41e5-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:40:47.951939 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.951954    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:40:47.952259 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.951962    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6111935-17ca-4c68-a361-740869af41e5-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:40:47.952259 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:47.951971    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cgsg\" (UniqueName: \"kubernetes.io/projected/e6111935-17ca-4c68-a361-740869af41e5-kube-api-access-7cgsg\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:40:48.645653 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.645611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk" event={"ID":"229e4f3b-dac2-4182-bb2d-96fef9573535","Type":"ContainerDied","Data":"073f0d971179bcb1af2d31b44443938cb44ebcec64a4122c37788302cd97224a"}
Jul 09 18:40:48.645875 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.645683    2578 scope.go:117] "RemoveContainer" containerID="70e9590e680e146c2c00fe4624b6423f739677a5cb12dd98b6083be69c2f6aa8"
Jul 09 18:40:48.645875 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.645624    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"
Jul 09 18:40:48.647706 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.647679    2578 generic.go:356] "Generic (PLEG): container finished" podID="e6111935-17ca-4c68-a361-740869af41e5" containerID="3af432e30338188e6582755d4aaa4e5f332c338f9b30b536a80410d4d5a1a18a" exitCode=0
Jul 09 18:40:48.647833 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.647747    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"
Jul 09 18:40:48.647833 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.647763    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" event={"ID":"e6111935-17ca-4c68-a361-740869af41e5","Type":"ContainerDied","Data":"3af432e30338188e6582755d4aaa4e5f332c338f9b30b536a80410d4d5a1a18a"}
Jul 09 18:40:48.647939 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.647839    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm" event={"ID":"e6111935-17ca-4c68-a361-740869af41e5","Type":"ContainerDied","Data":"1b35d974cd2ef19f938c638b43229fbc910f744d094bdb0e98e77f38a7bf7ed7"}
Jul 09 18:40:48.655730 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.655708    2578 scope.go:117] "RemoveContainer" containerID="4984dffc8cb64e109aa321f0f98cd821be42fa0b9d1b753731c90fb105297e00"
Jul 09 18:40:48.664753 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.664729    2578 scope.go:117] "RemoveContainer" containerID="08785b20c6b65b00f472978fcd5f6f6f7e24eb8cc40ec6157a663e654aaa35a1"
Jul 09 18:40:48.673241 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.673220    2578 scope.go:117] "RemoveContainer" containerID="e5fc516eac94018bdceccc6b7e6af4c110fa1c2cab31815ab6c378a509258d69"
Jul 09 18:40:48.682006 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.681987    2578 scope.go:117] "RemoveContainer" containerID="3af432e30338188e6582755d4aaa4e5f332c338f9b30b536a80410d4d5a1a18a"
Jul 09 18:40:48.685077 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.685053    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"]
Jul 09 18:40:48.690398 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.690379    2578 scope.go:117] "RemoveContainer" containerID="08785b20c6b65b00f472978fcd5f6f6f7e24eb8cc40ec6157a663e654aaa35a1"
Jul 09 18:40:48.690779 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:40:48.690753    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08785b20c6b65b00f472978fcd5f6f6f7e24eb8cc40ec6157a663e654aaa35a1\": container with ID starting with 08785b20c6b65b00f472978fcd5f6f6f7e24eb8cc40ec6157a663e654aaa35a1 not found: ID does not exist" containerID="08785b20c6b65b00f472978fcd5f6f6f7e24eb8cc40ec6157a663e654aaa35a1"
Jul 09 18:40:48.690843 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.690787    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08785b20c6b65b00f472978fcd5f6f6f7e24eb8cc40ec6157a663e654aaa35a1"} err="failed to get container status \"08785b20c6b65b00f472978fcd5f6f6f7e24eb8cc40ec6157a663e654aaa35a1\": rpc error: code = NotFound desc = could not find container \"08785b20c6b65b00f472978fcd5f6f6f7e24eb8cc40ec6157a663e654aaa35a1\": container with ID starting with 08785b20c6b65b00f472978fcd5f6f6f7e24eb8cc40ec6157a663e654aaa35a1 not found: ID does not exist"
Jul 09 18:40:48.690843 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.690826    2578 scope.go:117] "RemoveContainer" containerID="e5fc516eac94018bdceccc6b7e6af4c110fa1c2cab31815ab6c378a509258d69"
Jul 09 18:40:48.691119 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:40:48.691099    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5fc516eac94018bdceccc6b7e6af4c110fa1c2cab31815ab6c378a509258d69\": container with ID starting with e5fc516eac94018bdceccc6b7e6af4c110fa1c2cab31815ab6c378a509258d69 not found: ID does not exist" containerID="e5fc516eac94018bdceccc6b7e6af4c110fa1c2cab31815ab6c378a509258d69"
Jul 09 18:40:48.691160 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.691126    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5fc516eac94018bdceccc6b7e6af4c110fa1c2cab31815ab6c378a509258d69"} err="failed to get container status \"e5fc516eac94018bdceccc6b7e6af4c110fa1c2cab31815ab6c378a509258d69\": rpc error: code = NotFound desc = could not find container \"e5fc516eac94018bdceccc6b7e6af4c110fa1c2cab31815ab6c378a509258d69\": container with ID starting with e5fc516eac94018bdceccc6b7e6af4c110fa1c2cab31815ab6c378a509258d69 not found: ID does not exist"
Jul 09 18:40:48.691160 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.691143    2578 scope.go:117] "RemoveContainer" containerID="3af432e30338188e6582755d4aaa4e5f332c338f9b30b536a80410d4d5a1a18a"
Jul 09 18:40:48.691372 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:40:48.691358    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af432e30338188e6582755d4aaa4e5f332c338f9b30b536a80410d4d5a1a18a\": container with ID starting with 3af432e30338188e6582755d4aaa4e5f332c338f9b30b536a80410d4d5a1a18a not found: ID does not exist" containerID="3af432e30338188e6582755d4aaa4e5f332c338f9b30b536a80410d4d5a1a18a"
Jul 09 18:40:48.691423 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.691378    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af432e30338188e6582755d4aaa4e5f332c338f9b30b536a80410d4d5a1a18a"} err="failed to get container status \"3af432e30338188e6582755d4aaa4e5f332c338f9b30b536a80410d4d5a1a18a\": rpc error: code = NotFound desc = could not find container \"3af432e30338188e6582755d4aaa4e5f332c338f9b30b536a80410d4d5a1a18a\": container with ID starting with 3af432e30338188e6582755d4aaa4e5f332c338f9b30b536a80410d4d5a1a18a not found: ID does not exist"
Jul 09 18:40:48.694964 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.694937    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm"]
Jul 09 18:40:48.717330 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.717299    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"]
Jul 09 18:40:48.730646 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:48.730567    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk"]
Jul 09 18:40:49.402729 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:49.402696    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" path="/var/lib/kubelet/pods/229e4f3b-dac2-4182-bb2d-96fef9573535/volumes"
Jul 09 18:40:49.403163 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:49.403148    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6111935-17ca-4c68-a361-740869af41e5" path="/var/lib/kubelet/pods/e6111935-17ca-4c68-a361-740869af41e5/volumes"
Jul 09 18:40:59.243550 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.243514    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"]
Jul 09 18:40:59.244073 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.243954    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="llm-d-routing-sidecar"
Jul 09 18:40:59.244073 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.243973    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="llm-d-routing-sidecar"
Jul 09 18:40:59.244073 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.243998    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main"
Jul 09 18:40:59.244073 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.244007    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main"
Jul 09 18:40:59.244073 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.244026    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="storage-initializer"
Jul 09 18:40:59.244073 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.244034    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="storage-initializer"
Jul 09 18:40:59.244073 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.244050    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main"
Jul 09 18:40:59.244073 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.244057    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main"
Jul 09 18:40:59.244073 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.244069    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="storage-initializer"
Jul 09 18:40:59.244073 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.244078    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="storage-initializer"
Jul 09 18:40:59.244409 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.244158    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="main"
Jul 09 18:40:59.244409 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.244168    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6111935-17ca-4c68-a361-740869af41e5" containerName="llm-d-routing-sidecar"
Jul 09 18:40:59.244409 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.244179    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="229e4f3b-dac2-4182-bb2d-96fef9573535" containerName="main"
Jul 09 18:40:59.249707 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.249681    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.253123 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.253078    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-t8p8n\""
Jul 09 18:40:59.254251 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.254108    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 09 18:40:59.254251 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.254145    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:40:59.255434 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.255409    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"]
Jul 09 18:40:59.258939 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.258917    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.267213 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.267187    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"]
Jul 09 18:40:59.274257 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.274228    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"]
Jul 09 18:40:59.300048 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.300009    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:40:59.356457 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.356420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a147eb8a-1ff3-461e-8341-896a4b678e8c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.356457 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.356461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6dgd\" (UniqueName: \"kubernetes.io/projected/a147eb8a-1ff3-461e-8341-896a4b678e8c-kube-api-access-l6dgd\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.356683 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.356517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-dshm\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.356683 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.356570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.356683 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.356594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5m8m\" (UniqueName: \"kubernetes.io/projected/9f400574-a705-4172-a191-d24be3edaf8c-kube-api-access-q5m8m\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.356683 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.356641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f400574-a705-4172-a191-d24be3edaf8c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.356683 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.356658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.356683 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.356676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.356924 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.356728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.356924 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.356751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-home\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.356924 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.356819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.356924 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.356888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.357049 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.356932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.357049 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.356949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.457357 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.457311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.457560 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.457381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.457560 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.457404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.457560 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.457430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a147eb8a-1ff3-461e-8341-896a4b678e8c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.457560 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.457461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6dgd\" (UniqueName: \"kubernetes.io/projected/a147eb8a-1ff3-461e-8341-896a4b678e8c-kube-api-access-l6dgd\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.457839 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.457635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-dshm\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.457839 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.457669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.457839 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.457728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.457839 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.457732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.457839 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.457835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5m8m\" (UniqueName: \"kubernetes.io/projected/9f400574-a705-4172-a191-d24be3edaf8c-kube-api-access-q5m8m\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.458103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.457938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f400574-a705-4172-a191-d24be3edaf8c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.458103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.457970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.458103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.457991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.458103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.458013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.458103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.458044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.458103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.458093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-home\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.458414 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.458118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.458414 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.458269    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.458414 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.458311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.458414 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.458348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.458564 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.458465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.458564 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.458533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-home\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.459902 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.459877    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.460366 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.460343    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a147eb8a-1ff3-461e-8341-896a4b678e8c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.460519 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.460499    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-dshm\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.460673 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.460656    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f400574-a705-4172-a191-d24be3edaf8c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.468864 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.468838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5m8m\" (UniqueName: \"kubernetes.io/projected/9f400574-a705-4172-a191-d24be3edaf8c-kube-api-access-q5m8m\") pod \"custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.469108 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.469085    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6dgd\" (UniqueName: \"kubernetes.io/projected/a147eb8a-1ff3-461e-8341-896a4b678e8c-kube-api-access-l6dgd\") pod \"custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.525190 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.525101    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:40:59.561299 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.561258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:40:59.569805 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.569772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:40:59.728783 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:40:59.728751    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda147eb8a_1ff3_461e_8341_896a4b678e8c.slice/crio-82fa3f7ef10db17c2394e80ec9c257a3bb6e7d7111be2a8a4657c679ec084040 WatchSource:0}: Error finding container 82fa3f7ef10db17c2394e80ec9c257a3bb6e7d7111be2a8a4657c679ec084040: Status 404 returned error can't find the container with id 82fa3f7ef10db17c2394e80ec9c257a3bb6e7d7111be2a8a4657c679ec084040
Jul 09 18:40:59.730245 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.730221    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"]
Jul 09 18:40:59.756235 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:40:59.756203    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"]
Jul 09 18:40:59.765010 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:40:59.764980    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f400574_a705_4172_a191_d24be3edaf8c.slice/crio-7c1f72fd31db14de40ee7f6e7529af0ace792b948bf7b1f9c88d7132d92e1e5f WatchSource:0}: Error finding container 7c1f72fd31db14de40ee7f6e7529af0ace792b948bf7b1f9c88d7132d92e1e5f: Status 404 returned error can't find the container with id 7c1f72fd31db14de40ee7f6e7529af0ace792b948bf7b1f9c88d7132d92e1e5f
Jul 09 18:41:00.708465 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:00.708350    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" event={"ID":"a147eb8a-1ff3-461e-8341-896a4b678e8c","Type":"ContainerStarted","Data":"a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b"}
Jul 09 18:41:00.708465 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:00.708429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" event={"ID":"a147eb8a-1ff3-461e-8341-896a4b678e8c","Type":"ContainerStarted","Data":"82fa3f7ef10db17c2394e80ec9c257a3bb6e7d7111be2a8a4657c679ec084040"}
Jul 09 18:41:00.709565 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:00.708607    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:41:00.710593 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:00.710538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" event={"ID":"9f400574-a705-4172-a191-d24be3edaf8c","Type":"ContainerStarted","Data":"847b440e14b6e73e6b20e1708ebfa507cf50a67e1a82eaaa5e44aa51e32a7d43"}
Jul 09 18:41:00.710745 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:00.710608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" event={"ID":"9f400574-a705-4172-a191-d24be3edaf8c","Type":"ContainerStarted","Data":"7c1f72fd31db14de40ee7f6e7529af0ace792b948bf7b1f9c88d7132d92e1e5f"}
Jul 09 18:41:01.719045 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:01.718703    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" event={"ID":"a147eb8a-1ff3-461e-8341-896a4b678e8c","Type":"ContainerStarted","Data":"a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d"}
Jul 09 18:41:03.736334 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:03.736279    2578 generic.go:356] "Generic (PLEG): container finished" podID="9f400574-a705-4172-a191-d24be3edaf8c" containerID="847b440e14b6e73e6b20e1708ebfa507cf50a67e1a82eaaa5e44aa51e32a7d43" exitCode=0
Jul 09 18:41:03.737181 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:03.736345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" event={"ID":"9f400574-a705-4172-a191-d24be3edaf8c","Type":"ContainerDied","Data":"847b440e14b6e73e6b20e1708ebfa507cf50a67e1a82eaaa5e44aa51e32a7d43"}
Jul 09 18:41:04.743730 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:04.743636    2578 generic.go:356] "Generic (PLEG): container finished" podID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerID="a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d" exitCode=0
Jul 09 18:41:04.744166 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:04.743715    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" event={"ID":"a147eb8a-1ff3-461e-8341-896a4b678e8c","Type":"ContainerDied","Data":"a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d"}
Jul 09 18:41:04.746188 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:04.745903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" event={"ID":"9f400574-a705-4172-a191-d24be3edaf8c","Type":"ContainerStarted","Data":"aff0d79e5effc0b1e899402c0702297db05d7230c29f7ca390ed357557b57532"}
Jul 09 18:41:04.796206 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:04.796135    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podStartSLOduration=2.415934913 podStartE2EDuration="5.796112269s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.380177356s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:41:04.794236408 +0000 UTC m=+2187.981214200" watchObservedRunningTime="2026-07-09 18:41:04.796112269 +0000 UTC m=+2187.983090038"
Jul 09 18:41:05.754827 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:05.754771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" event={"ID":"a147eb8a-1ff3-461e-8341-896a4b678e8c","Type":"ContainerStarted","Data":"00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0"}
Jul 09 18:41:05.782442 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:05.782377    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podStartSLOduration=3.117903048 podStartE2EDuration="6.782355899s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.664452851s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:41:05.78023171 +0000 UTC m=+2188.967209539" watchObservedRunningTime="2026-07-09 18:41:05.782355899 +0000 UTC m=+2188.969333668"
Jul 09 18:41:09.562000 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:09.561950    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:41:09.562525 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:09.562041    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:41:09.563401 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:09.563364    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:41:09.570614 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:09.570576    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:41:09.570787 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:09.570645    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:41:09.572099 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:09.572067    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:41:09.575626 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:09.575598    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:41:19.562409 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:19.562361    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:41:19.571023 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:19.570975    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:41:20.573764 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:20.573728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:41:27.463407 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:27.463363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:41:27.563330 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:27.563290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:41:29.561689 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:29.561644    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:41:29.571024 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:29.570976    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:41:39.562681 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:39.562562    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:41:39.570681 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:39.570635    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:41:49.562286 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:49.562233    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:41:49.570918 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:49.570873    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:41:53.355977 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:41:53.355925    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 18:41:54.015442 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:54.015409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:41:59.562353 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:59.562236    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:41:59.571201 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:41:59.571156    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:42:09.562466 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:42:09.562411    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:42:09.570790 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:42:09.570737    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:42:19.561982 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:42:19.561938    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:42:19.570495 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:42:19.570451    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:42:29.562255 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:42:29.562205    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:42:29.570434 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:42:29.570395    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:42:39.561969 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:42:39.561907    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:42:39.571325 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:42:39.571282    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:42:49.562302 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:42:49.562250    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:42:49.570900 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:42:49.570868    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:42:59.562065 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:42:59.562019    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:42:59.571178 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:42:59.571139    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:43:09.562646 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:09.562595    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:43:09.570824 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:09.570764    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:43:19.561908 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:19.561859    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:43:19.570512 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:19.570471    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:43:29.562148 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:29.562030    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:43:29.571372 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:29.571270    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:43:34.968982 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:34.968943    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:43:36.181383 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:36.181345    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:43:39.562149 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:39.562097    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:43:39.570879 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:39.570836    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:43:49.562591 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:49.562536    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 09 18:43:49.570331 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:49.570287    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 09 18:43:57.016949 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:43:57.016890    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 18:43:57.619009 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:57.618974    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:43:59.573073 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:59.573036    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:43:59.581438 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:59.581409    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:43:59.589819 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:59.589772    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:43:59.640555 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:43:59.640525    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:44:26.541348 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:26.541292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:44:26.541836 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:26.541424    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"]
Jul 09 18:44:26.545412 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:26.545381    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"]
Jul 09 18:44:26.602916 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:26.602880    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:44:37.435305 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:37.435274    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:44:37.441668 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:37.441634    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:44:37.443650 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:37.443626    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:44:37.449937 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:37.449910    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:44:41.566808 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:41.566735    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main" containerID="cri-o://00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0" gracePeriod=45
Jul 09 18:44:41.573875 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:41.573831    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main" containerID="cri-o://aff0d79e5effc0b1e899402c0702297db05d7230c29f7ca390ed357557b57532" gracePeriod=45
Jul 09 18:44:42.392554 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.392517    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="llm-d-routing-sidecar" containerID="cri-o://a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b" gracePeriod=45
Jul 09 18:44:42.527893 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.527864    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:44:42.595534 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.595442    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-home\") pod \"9f400574-a705-4172-a191-d24be3edaf8c\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") "
Jul 09 18:44:42.595534 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.595523    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5m8m\" (UniqueName: \"kubernetes.io/projected/9f400574-a705-4172-a191-d24be3edaf8c-kube-api-access-q5m8m\") pod \"9f400574-a705-4172-a191-d24be3edaf8c\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") "
Jul 09 18:44:42.596084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.595547    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-model-cache\") pod \"9f400574-a705-4172-a191-d24be3edaf8c\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") "
Jul 09 18:44:42.596084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.595590    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-kserve-provision-location\") pod \"9f400574-a705-4172-a191-d24be3edaf8c\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") "
Jul 09 18:44:42.596084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.595732    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-tmp-dir\") pod \"9f400574-a705-4172-a191-d24be3edaf8c\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") "
Jul 09 18:44:42.596084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.595786    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f400574-a705-4172-a191-d24be3edaf8c-tls-certs\") pod \"9f400574-a705-4172-a191-d24be3edaf8c\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") "
Jul 09 18:44:42.596084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.595845    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-dshm\") pod \"9f400574-a705-4172-a191-d24be3edaf8c\" (UID: \"9f400574-a705-4172-a191-d24be3edaf8c\") "
Jul 09 18:44:42.596084 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.595868    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f400574-a705-4172-a191-d24be3edaf8c" (UID: "9f400574-a705-4172-a191-d24be3edaf8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:42.596356 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.596173    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:44:42.596356 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.596244    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-home" (OuterVolumeSpecName: "home") pod "9f400574-a705-4172-a191-d24be3edaf8c" (UID: "9f400574-a705-4172-a191-d24be3edaf8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:42.598407 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.598363    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f400574-a705-4172-a191-d24be3edaf8c-kube-api-access-q5m8m" (OuterVolumeSpecName: "kube-api-access-q5m8m") pod "9f400574-a705-4172-a191-d24be3edaf8c" (UID: "9f400574-a705-4172-a191-d24be3edaf8c"). InnerVolumeSpecName "kube-api-access-q5m8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:44:42.598407 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.598388    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-dshm" (OuterVolumeSpecName: "dshm") pod "9f400574-a705-4172-a191-d24be3edaf8c" (UID: "9f400574-a705-4172-a191-d24be3edaf8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:42.598549 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.598476    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f400574-a705-4172-a191-d24be3edaf8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f400574-a705-4172-a191-d24be3edaf8c" (UID: "9f400574-a705-4172-a191-d24be3edaf8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:44:42.611189 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.611137    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f400574-a705-4172-a191-d24be3edaf8c" (UID: "9f400574-a705-4172-a191-d24be3edaf8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:42.652273 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.652246    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:44:42.674742 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.674696    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f400574-a705-4172-a191-d24be3edaf8c" (UID: "9f400574-a705-4172-a191-d24be3edaf8c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:42.697456 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.697421    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-kserve-provision-location\") pod \"a147eb8a-1ff3-461e-8341-896a4b678e8c\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") "
Jul 09 18:44:42.697657 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.697468    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-tmp-dir\") pod \"a147eb8a-1ff3-461e-8341-896a4b678e8c\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") "
Jul 09 18:44:42.697657 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.697496    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a147eb8a-1ff3-461e-8341-896a4b678e8c-tls-certs\") pod \"a147eb8a-1ff3-461e-8341-896a4b678e8c\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") "
Jul 09 18:44:42.697657 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.697510    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-model-cache\") pod \"a147eb8a-1ff3-461e-8341-896a4b678e8c\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") "
Jul 09 18:44:42.697657 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.697570    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-home\") pod \"a147eb8a-1ff3-461e-8341-896a4b678e8c\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") "
Jul 09 18:44:42.697657 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.697603    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6dgd\" (UniqueName: \"kubernetes.io/projected/a147eb8a-1ff3-461e-8341-896a4b678e8c-kube-api-access-l6dgd\") pod \"a147eb8a-1ff3-461e-8341-896a4b678e8c\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") "
Jul 09 18:44:42.697940 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.697734    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-dshm\") pod \"a147eb8a-1ff3-461e-8341-896a4b678e8c\" (UID: \"a147eb8a-1ff3-461e-8341-896a4b678e8c\") "
Jul 09 18:44:42.698128 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.698102    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5m8m\" (UniqueName: \"kubernetes.io/projected/9f400574-a705-4172-a191-d24be3edaf8c-kube-api-access-q5m8m\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:44:42.698216 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.698135    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:44:42.698216 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.698148    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:44:42.698216 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.698162    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f400574-a705-4172-a191-d24be3edaf8c-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:44:42.698216 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.698175    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:44:42.698216 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.698186    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f400574-a705-4172-a191-d24be3edaf8c-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:44:42.698468 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.698327    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "a147eb8a-1ff3-461e-8341-896a4b678e8c" (UID: "a147eb8a-1ff3-461e-8341-896a4b678e8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:42.698739 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.698700    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-home" (OuterVolumeSpecName: "home") pod "a147eb8a-1ff3-461e-8341-896a4b678e8c" (UID: "a147eb8a-1ff3-461e-8341-896a4b678e8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:42.700144 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.700110    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a147eb8a-1ff3-461e-8341-896a4b678e8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a147eb8a-1ff3-461e-8341-896a4b678e8c" (UID: "a147eb8a-1ff3-461e-8341-896a4b678e8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:44:42.700285 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.700254    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-dshm" (OuterVolumeSpecName: "dshm") pod "a147eb8a-1ff3-461e-8341-896a4b678e8c" (UID: "a147eb8a-1ff3-461e-8341-896a4b678e8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:42.700563 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.700352    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a147eb8a-1ff3-461e-8341-896a4b678e8c-kube-api-access-l6dgd" (OuterVolumeSpecName: "kube-api-access-l6dgd") pod "a147eb8a-1ff3-461e-8341-896a4b678e8c" (UID: "a147eb8a-1ff3-461e-8341-896a4b678e8c"). InnerVolumeSpecName "kube-api-access-l6dgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:44:42.712332 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.712287    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a147eb8a-1ff3-461e-8341-896a4b678e8c" (UID: "a147eb8a-1ff3-461e-8341-896a4b678e8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:42.761032 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.760984    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a147eb8a-1ff3-461e-8341-896a4b678e8c" (UID: "a147eb8a-1ff3-461e-8341-896a4b678e8c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:42.799553 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.799517    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:44:42.799553 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.799552    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6dgd\" (UniqueName: \"kubernetes.io/projected/a147eb8a-1ff3-461e-8341-896a4b678e8c-kube-api-access-l6dgd\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:44:42.799553 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.799562    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:44:42.799846 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.799571    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:44:42.799846 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.799579    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:44:42.799846 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.799588    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a147eb8a-1ff3-461e-8341-896a4b678e8c-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:44:42.799846 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.799596    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a147eb8a-1ff3-461e-8341-896a4b678e8c-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:44:42.821932 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.821899    2578 generic.go:356] "Generic (PLEG): container finished" podID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerID="00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0" exitCode=0
Jul 09 18:44:42.821932 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.821927    2578 generic.go:356] "Generic (PLEG): container finished" podID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerID="a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b" exitCode=0
Jul 09 18:44:42.822146 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.821953    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"
Jul 09 18:44:42.822146 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.821980    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" event={"ID":"a147eb8a-1ff3-461e-8341-896a4b678e8c","Type":"ContainerDied","Data":"00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0"}
Jul 09 18:44:42.822146 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.822033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" event={"ID":"a147eb8a-1ff3-461e-8341-896a4b678e8c","Type":"ContainerDied","Data":"a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b"}
Jul 09 18:44:42.822146 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.822051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4" event={"ID":"a147eb8a-1ff3-461e-8341-896a4b678e8c","Type":"ContainerDied","Data":"82fa3f7ef10db17c2394e80ec9c257a3bb6e7d7111be2a8a4657c679ec084040"}
Jul 09 18:44:42.822146 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.822081    2578 scope.go:117] "RemoveContainer" containerID="00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0"
Jul 09 18:44:42.823605 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.823583    2578 generic.go:356] "Generic (PLEG): container finished" podID="9f400574-a705-4172-a191-d24be3edaf8c" containerID="aff0d79e5effc0b1e899402c0702297db05d7230c29f7ca390ed357557b57532" exitCode=0
Jul 09 18:44:42.823753 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.823640    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"
Jul 09 18:44:42.823753 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.823659    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" event={"ID":"9f400574-a705-4172-a191-d24be3edaf8c","Type":"ContainerDied","Data":"aff0d79e5effc0b1e899402c0702297db05d7230c29f7ca390ed357557b57532"}
Jul 09 18:44:42.823753 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.823724    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp" event={"ID":"9f400574-a705-4172-a191-d24be3edaf8c","Type":"ContainerDied","Data":"7c1f72fd31db14de40ee7f6e7529af0ace792b948bf7b1f9c88d7132d92e1e5f"}
Jul 09 18:44:42.832418 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.832249    2578 scope.go:117] "RemoveContainer" containerID="a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d"
Jul 09 18:44:42.840763 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.840740    2578 scope.go:117] "RemoveContainer" containerID="a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b"
Jul 09 18:44:42.848691 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.848666    2578 scope.go:117] "RemoveContainer" containerID="00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0"
Jul 09 18:44:42.849240 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:44:42.849085    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0\": container with ID starting with 00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0 not found: ID does not exist" containerID="00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0"
Jul 09 18:44:42.849240 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.849120    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0"} err="failed to get container status \"00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0\": rpc error: code = NotFound desc = could not find container \"00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0\": container with ID starting with 00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0 not found: ID does not exist"
Jul 09 18:44:42.849240 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.849144    2578 scope.go:117] "RemoveContainer" containerID="a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d"
Jul 09 18:44:42.850150 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:44:42.850122    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d\": container with ID starting with a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d not found: ID does not exist" containerID="a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d"
Jul 09 18:44:42.850249 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.850157    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d"} err="failed to get container status \"a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d\": rpc error: code = NotFound desc = could not find container \"a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d\": container with ID starting with a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d not found: ID does not exist"
Jul 09 18:44:42.850249 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.850178    2578 scope.go:117] "RemoveContainer" containerID="a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b"
Jul 09 18:44:42.850444 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:44:42.850424    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b\": container with ID starting with a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b not found: ID does not exist" containerID="a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b"
Jul 09 18:44:42.850505 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.850450    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b"} err="failed to get container status \"a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b\": rpc error: code = NotFound desc = could not find container \"a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b\": container with ID starting with a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b not found: ID does not exist"
Jul 09 18:44:42.850505 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.850468    2578 scope.go:117] "RemoveContainer" containerID="00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0"
Jul 09 18:44:42.850577 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.850554    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"]
Jul 09 18:44:42.850764 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.850739    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0"} err="failed to get container status \"00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0\": rpc error: code = NotFound desc = could not find container \"00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0\": container with ID starting with 00fd96a584b93d2bfca605cbde49d8308150e34f75b8aa51b5883c84360377e0 not found: ID does not exist"
Jul 09 18:44:42.850836 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.850766    2578 scope.go:117] "RemoveContainer" containerID="a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d"
Jul 09 18:44:42.851059 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.851039    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d"} err="failed to get container status \"a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d\": rpc error: code = NotFound desc = could not find container \"a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d\": container with ID starting with a4028fa368c37bcc8807e66285e703bc1159ef6268c822302b5b8337b874330d not found: ID does not exist"
Jul 09 18:44:42.851125 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.851059    2578 scope.go:117] "RemoveContainer" containerID="a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b"
Jul 09 18:44:42.851294 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.851275    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b"} err="failed to get container status \"a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b\": rpc error: code = NotFound desc = could not find container \"a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b\": container with ID starting with a2c366d1234c2c6703b00e95b2a669863878c43607e383e26825c86f296dba5b not found: ID does not exist"
Jul 09 18:44:42.851368 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.851307    2578 scope.go:117] "RemoveContainer" containerID="aff0d79e5effc0b1e899402c0702297db05d7230c29f7ca390ed357557b57532"
Jul 09 18:44:42.855007 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.854980    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp"]
Jul 09 18:44:42.859761 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.859742    2578 scope.go:117] "RemoveContainer" containerID="847b440e14b6e73e6b20e1708ebfa507cf50a67e1a82eaaa5e44aa51e32a7d43"
Jul 09 18:44:42.868035 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.867995    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"]
Jul 09 18:44:42.868107 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.868090    2578 scope.go:117] "RemoveContainer" containerID="aff0d79e5effc0b1e899402c0702297db05d7230c29f7ca390ed357557b57532"
Jul 09 18:44:42.868387 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:44:42.868371    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff0d79e5effc0b1e899402c0702297db05d7230c29f7ca390ed357557b57532\": container with ID starting with aff0d79e5effc0b1e899402c0702297db05d7230c29f7ca390ed357557b57532 not found: ID does not exist" containerID="aff0d79e5effc0b1e899402c0702297db05d7230c29f7ca390ed357557b57532"
Jul 09 18:44:42.868436 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.868392    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff0d79e5effc0b1e899402c0702297db05d7230c29f7ca390ed357557b57532"} err="failed to get container status \"aff0d79e5effc0b1e899402c0702297db05d7230c29f7ca390ed357557b57532\": rpc error: code = NotFound desc = could not find container \"aff0d79e5effc0b1e899402c0702297db05d7230c29f7ca390ed357557b57532\": container with ID starting with aff0d79e5effc0b1e899402c0702297db05d7230c29f7ca390ed357557b57532 not found: ID does not exist"
Jul 09 18:44:42.868436 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.868411    2578 scope.go:117] "RemoveContainer" containerID="847b440e14b6e73e6b20e1708ebfa507cf50a67e1a82eaaa5e44aa51e32a7d43"
Jul 09 18:44:42.868664 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:44:42.868650    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847b440e14b6e73e6b20e1708ebfa507cf50a67e1a82eaaa5e44aa51e32a7d43\": container with ID starting with 847b440e14b6e73e6b20e1708ebfa507cf50a67e1a82eaaa5e44aa51e32a7d43 not found: ID does not exist" containerID="847b440e14b6e73e6b20e1708ebfa507cf50a67e1a82eaaa5e44aa51e32a7d43"
Jul 09 18:44:42.868705 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.868668    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847b440e14b6e73e6b20e1708ebfa507cf50a67e1a82eaaa5e44aa51e32a7d43"} err="failed to get container status \"847b440e14b6e73e6b20e1708ebfa507cf50a67e1a82eaaa5e44aa51e32a7d43\": rpc error: code = NotFound desc = could not find container \"847b440e14b6e73e6b20e1708ebfa507cf50a67e1a82eaaa5e44aa51e32a7d43\": container with ID starting with 847b440e14b6e73e6b20e1708ebfa507cf50a67e1a82eaaa5e44aa51e32a7d43 not found: ID does not exist"
Jul 09 18:44:42.874172 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:42.874148    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4"]
Jul 09 18:44:43.402890 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:43.402854    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f400574-a705-4172-a191-d24be3edaf8c" path="/var/lib/kubelet/pods/9f400574-a705-4172-a191-d24be3edaf8c/volumes"
Jul 09 18:44:43.403312 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:43.403299    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" path="/var/lib/kubelet/pods/a147eb8a-1ff3-461e-8341-896a4b678e8c/volumes"
Jul 09 18:44:47.972773 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:47.972733    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:44:48.067298 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:48.067257    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:44:51.335279 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.335239    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"]
Jul 09 18:44:51.335678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.335542    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="llm-d-routing-sidecar"
Jul 09 18:44:51.335678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.335553    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="llm-d-routing-sidecar"
Jul 09 18:44:51.335678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.335561    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main"
Jul 09 18:44:51.335678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.335567    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main"
Jul 09 18:44:51.335678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.335574    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="storage-initializer"
Jul 09 18:44:51.335678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.335581    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="storage-initializer"
Jul 09 18:44:51.335678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.335589    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="storage-initializer"
Jul 09 18:44:51.335678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.335594    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="storage-initializer"
Jul 09 18:44:51.335678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.335604    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main"
Jul 09 18:44:51.335678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.335610    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main"
Jul 09 18:44:51.335678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.335658    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f400574-a705-4172-a191-d24be3edaf8c" containerName="main"
Jul 09 18:44:51.335678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.335665    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="main"
Jul 09 18:44:51.335678 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.335673    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a147eb8a-1ff3-461e-8341-896a4b678e8c" containerName="llm-d-routing-sidecar"
Jul 09 18:44:51.340007 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.339986    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.343003 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.342975    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-q8vnq\""
Jul 09 18:44:51.343156 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.343103    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 09 18:44:51.350050 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.350021    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"]
Jul 09 18:44:51.475969 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.475932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.475969 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.475977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.476195 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.475995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.476195 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.476056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a89107-00ea-4275-ab07-7291b5b7e3c4-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.476195 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.476093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a89107-00ea-4275-ab07-7291b5b7e3c4-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.476195 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.476139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g972t\" (UniqueName: \"kubernetes.io/projected/44a89107-00ea-4275-ab07-7291b5b7e3c4-kube-api-access-g972t\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.476195 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.476162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a89107-00ea-4275-ab07-7291b5b7e3c4-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.476360 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.476202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.476360 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.476240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.577247 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.577210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g972t\" (UniqueName: \"kubernetes.io/projected/44a89107-00ea-4275-ab07-7291b5b7e3c4-kube-api-access-g972t\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.577247 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.577249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a89107-00ea-4275-ab07-7291b5b7e3c4-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.577469 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.577284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.577469 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.577315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.577469 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.577344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.577469 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.577373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.577469 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.577395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.577698 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.577663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a89107-00ea-4275-ab07-7291b5b7e3c4-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.577757 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.577716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a89107-00ea-4275-ab07-7291b5b7e3c4-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.577855 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.577777    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.577917 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.577903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.577974 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.577926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.578014 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.577973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.578445 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.578422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a89107-00ea-4275-ab07-7291b5b7e3c4-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.579806 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.579769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a89107-00ea-4275-ab07-7291b5b7e3c4-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.579931 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.579914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a89107-00ea-4275-ab07-7291b5b7e3c4-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.586766 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.586662    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a89107-00ea-4275-ab07-7291b5b7e3c4-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.586906 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.586838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g972t\" (UniqueName: \"kubernetes.io/projected/44a89107-00ea-4275-ab07-7291b5b7e3c4-kube-api-access-g972t\") pod \"router-gateway-2-openshift-default-684dc9b4b9-r8qw5\" (UID: \"44a89107-00ea-4275-ab07-7291b5b7e3c4\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.651217 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.651166    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:51.795057 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.795008    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"]
Jul 09 18:44:51.797384 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:44:51.797352    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a89107_00ea_4275_ab07_7291b5b7e3c4.slice/crio-1af9e4c50ba9b7c8a909fa553dee444351a7e4eb8a2a85fed4cc87b8581b4f5b WatchSource:0}: Error finding container 1af9e4c50ba9b7c8a909fa553dee444351a7e4eb8a2a85fed4cc87b8581b4f5b: Status 404 returned error can't find the container with id 1af9e4c50ba9b7c8a909fa553dee444351a7e4eb8a2a85fed4cc87b8581b4f5b
Jul 09 18:44:51.799626 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.799590    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:44:51.799741 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.799673    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:44:51.799741 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.799717    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:44:51.866905 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.866861    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5" event={"ID":"44a89107-00ea-4275-ab07-7291b5b7e3c4","Type":"ContainerStarted","Data":"aae98ca7ea8951a481a0eebd7cafb06bd22925f4258b0d2ee24931efa1e3655d"}
Jul 09 18:44:51.867065 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.866923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5" event={"ID":"44a89107-00ea-4275-ab07-7291b5b7e3c4","Type":"ContainerStarted","Data":"1af9e4c50ba9b7c8a909fa553dee444351a7e4eb8a2a85fed4cc87b8581b4f5b"}
Jul 09 18:44:51.889358 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:51.889198    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5" podStartSLOduration=0.889008223 podStartE2EDuration="889.008223ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:44:51.88786386 +0000 UTC m=+2415.074841628" watchObservedRunningTime="2026-07-09 18:44:51.889008223 +0000 UTC m=+2415.075985993"
Jul 09 18:44:52.025710 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:52.025672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:44:52.651358 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:52.651318    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:52.656460 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:52.656428    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:52.872058 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:52.872027    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:44:52.873161 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:44:52.873142    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:45:02.916942 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:02.916905    2578 generic.go:356] "Generic (PLEG): container finished" podID="44a89107-00ea-4275-ab07-7291b5b7e3c4" containerID="aae98ca7ea8951a481a0eebd7cafb06bd22925f4258b0d2ee24931efa1e3655d" exitCode=0
Jul 09 18:45:02.917372 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:02.916985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5" event={"ID":"44a89107-00ea-4275-ab07-7291b5b7e3c4","Type":"ContainerDied","Data":"aae98ca7ea8951a481a0eebd7cafb06bd22925f4258b0d2ee24931efa1e3655d"}
Jul 09 18:45:02.917422 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:02.917389    2578 scope.go:117] "RemoveContainer" containerID="aae98ca7ea8951a481a0eebd7cafb06bd22925f4258b0d2ee24931efa1e3655d"
Jul 09 18:45:02.918382 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:02.918353    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:45:02.918497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:02.918438    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:45:02.918497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:02.918487    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:45:03.926843 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:03.926774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5" event={"ID":"44a89107-00ea-4275-ab07-7291b5b7e3c4","Type":"ContainerStarted","Data":"9a65ff41377e0e331f7b94c228824a79dc8c553b50e474db45d016a4b958d56b"}
Jul 09 18:45:04.652155 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:04.652124    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:45:04.653403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:04.653372    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5" podUID="44a89107-00ea-4275-ab07-7291b5b7e3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Jul 09 18:45:05.651554 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:05.651515    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5" podUID="44a89107-00ea-4275-ab07-7291b5b7e3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Jul 09 18:45:06.651974 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:06.651911    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5" podUID="44a89107-00ea-4275-ab07-7291b5b7e3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Jul 09 18:45:07.652381 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:07.652332    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5" podUID="44a89107-00ea-4275-ab07-7291b5b7e3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Jul 09 18:45:07.872557 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:07.872520    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:45:08.652530 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:08.652486    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5" podUID="44a89107-00ea-4275-ab07-7291b5b7e3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Jul 09 18:45:09.656033 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:09.655998    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:45:09.657396 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:09.657375    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5"
Jul 09 18:45:13.003046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.003008    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"]
Jul 09 18:45:13.005292 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.005273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.009650 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.009625    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-62lxz\""
Jul 09 18:45:13.009954 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.009935    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:45:13.009954 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.009944    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 09 18:45:13.024883 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.024845    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"]
Jul 09 18:45:13.043183 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.043151    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"]
Jul 09 18:45:13.045734 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.045458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.063608 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.063576    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"]
Jul 09 18:45:13.080459 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.080407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:45:13.088989 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.088957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-model-cache\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.089157 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.089006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.089157 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.089025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.089157 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.089043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-dshm\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.089157 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.089059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.089157 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.089073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klpf9\" (UniqueName: \"kubernetes.io/projected/cd420cb5-4e2e-4536-979f-47fdd89b47e4-kube-api-access-klpf9\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.089157 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.089146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.089386 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.089190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd420cb5-4e2e-4536-979f-47fdd89b47e4-tls-certs\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.089386 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.089210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-home\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.089386 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.089235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-home\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.089386 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.089256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64b5b28-6f43-43f6-b09b-64b20519a92a-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.089386 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.089273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzsm8\" (UniqueName: \"kubernetes.io/projected/d64b5b28-6f43-43f6-b09b-64b20519a92a-kube-api-access-vzsm8\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.089386 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.089334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.089603 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.089386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.189814 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.189769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.190000 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.189857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd420cb5-4e2e-4536-979f-47fdd89b47e4-tls-certs\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.190000 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.189877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-home\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.190000 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.189901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-home\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.190000 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.189918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64b5b28-6f43-43f6-b09b-64b20519a92a-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.190000 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.189945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzsm8\" (UniqueName: \"kubernetes.io/projected/d64b5b28-6f43-43f6-b09b-64b20519a92a-kube-api-access-vzsm8\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.190284 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.190284 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.190284 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.190284 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-model-cache\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.190515 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190293    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-home\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.190515 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.190515 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-home\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.190515 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.190515 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-dshm\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.190515 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.190515 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpf9\" (UniqueName: \"kubernetes.io/projected/cd420cb5-4e2e-4536-979f-47fdd89b47e4-kube-api-access-klpf9\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.190515 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190480    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.190970 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190561    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-model-cache\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.190970 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190676    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.190970 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.190970 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.190782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.192685 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.192653    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-dshm\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.192853 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.192825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.192968 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.192948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd420cb5-4e2e-4536-979f-47fdd89b47e4-tls-certs\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.193026 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.193012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64b5b28-6f43-43f6-b09b-64b20519a92a-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.199589 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.199559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klpf9\" (UniqueName: \"kubernetes.io/projected/cd420cb5-4e2e-4536-979f-47fdd89b47e4-kube-api-access-klpf9\") pod \"router-with-refs-pd-test-kserve-6db46874d9-zt6ff\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.199714 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.199683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzsm8\" (UniqueName: \"kubernetes.io/projected/d64b5b28-6f43-43f6-b09b-64b20519a92a-kube-api-access-vzsm8\") pod \"router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.314996 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.314885    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:13.356449 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.356403    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:13.475772 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.475722    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"]
Jul 09 18:45:13.480404 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:45:13.480368    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd420cb5_4e2e_4536_979f_47fdd89b47e4.slice/crio-305c66666fd233e70e6333620c40f803e65da9d0876842df8e0185a0dfc040e8 WatchSource:0}: Error finding container 305c66666fd233e70e6333620c40f803e65da9d0876842df8e0185a0dfc040e8: Status 404 returned error can't find the container with id 305c66666fd233e70e6333620c40f803e65da9d0876842df8e0185a0dfc040e8
Jul 09 18:45:13.512176 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.512145    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"]
Jul 09 18:45:13.515870 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:45:13.515838    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64b5b28_6f43_43f6_b09b_64b20519a92a.slice/crio-6ad31879b53d4063966da569242686a435c66aa4ef5a61cd03bafab3fd90752d WatchSource:0}: Error finding container 6ad31879b53d4063966da569242686a435c66aa4ef5a61cd03bafab3fd90752d: Status 404 returned error can't find the container with id 6ad31879b53d4063966da569242686a435c66aa4ef5a61cd03bafab3fd90752d
Jul 09 18:45:13.763645 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.763602    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:45:13.865860 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.865820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:45:13.979739 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.979693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" event={"ID":"d64b5b28-6f43-43f6-b09b-64b20519a92a","Type":"ContainerStarted","Data":"40358f6d7d5639be55e398253813331f65bb55bed448879ad52ac6bf3c54406d"}
Jul 09 18:45:13.979983 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.979765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" event={"ID":"d64b5b28-6f43-43f6-b09b-64b20519a92a","Type":"ContainerStarted","Data":"6ad31879b53d4063966da569242686a435c66aa4ef5a61cd03bafab3fd90752d"}
Jul 09 18:45:13.981547 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.981514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" event={"ID":"cd420cb5-4e2e-4536-979f-47fdd89b47e4","Type":"ContainerStarted","Data":"68a05ad9d12613441a68018ff7566d5c0fd00ed1aa07299c8f1477cfbd4fb31e"}
Jul 09 18:45:13.981699 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.981572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" event={"ID":"cd420cb5-4e2e-4536-979f-47fdd89b47e4","Type":"ContainerStarted","Data":"305c66666fd233e70e6333620c40f803e65da9d0876842df8e0185a0dfc040e8"}
Jul 09 18:45:13.981699 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:13.981603    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:14.988509 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:14.988471    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" event={"ID":"cd420cb5-4e2e-4536-979f-47fdd89b47e4","Type":"ContainerStarted","Data":"db0e7e04283aaf52bb75db3418240ed7aa0f2e8aee99e9ac5f837d615e42af0d"}
Jul 09 18:45:18.009858 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:18.009790    2578 generic.go:356] "Generic (PLEG): container finished" podID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerID="40358f6d7d5639be55e398253813331f65bb55bed448879ad52ac6bf3c54406d" exitCode=0
Jul 09 18:45:18.010359 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:18.009868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" event={"ID":"d64b5b28-6f43-43f6-b09b-64b20519a92a","Type":"ContainerDied","Data":"40358f6d7d5639be55e398253813331f65bb55bed448879ad52ac6bf3c54406d"}
Jul 09 18:45:18.011769 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:18.011744    2578 generic.go:356] "Generic (PLEG): container finished" podID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerID="db0e7e04283aaf52bb75db3418240ed7aa0f2e8aee99e9ac5f837d615e42af0d" exitCode=0
Jul 09 18:45:18.011889 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:18.011779    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" event={"ID":"cd420cb5-4e2e-4536-979f-47fdd89b47e4","Type":"ContainerDied","Data":"db0e7e04283aaf52bb75db3418240ed7aa0f2e8aee99e9ac5f837d615e42af0d"}
Jul 09 18:45:19.020008 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:19.019966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" event={"ID":"cd420cb5-4e2e-4536-979f-47fdd89b47e4","Type":"ContainerStarted","Data":"73bfa8d06b5f844fcbaeec5fa3a3991b8d934356ca2f20db6c724d872b424a94"}
Jul 09 18:45:19.022100 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:19.022074    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" event={"ID":"d64b5b28-6f43-43f6-b09b-64b20519a92a","Type":"ContainerStarted","Data":"8f4ddd29a1f25b7326d4cc543767752d6378c3576a3dafa943ec4582e87f30eb"}
Jul 09 18:45:19.047442 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:19.047372    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podStartSLOduration=3.799282997 podStartE2EDuration="7.047350787s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.24806779s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:45:19.046412188 +0000 UTC m=+2442.233389960" watchObservedRunningTime="2026-07-09 18:45:19.047350787 +0000 UTC m=+2442.234328556"
Jul 09 18:45:19.071853 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:19.071765    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podStartSLOduration=2.373111254 podStartE2EDuration="6.07174246s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698631206s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:45:19.068956049 +0000 UTC m=+2442.255933843" watchObservedRunningTime="2026-07-09 18:45:19.07174246 +0000 UTC m=+2442.258720231"
Jul 09 18:45:23.030705 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:23.030661    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:45:23.315761 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:23.315655    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:23.315761 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:23.315727    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:23.317328 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:23.317294    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:45:23.330175 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:23.330138    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:45:23.356695 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:23.356661    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:23.356926 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:23.356725    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:45:23.358475 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:23.358441    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:45:33.315674 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:33.315619    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:45:33.357938 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:33.357882    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:45:43.316118 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:43.316055    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:45:43.357124 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:43.357073    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:45:53.316352 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:53.316307    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:45:53.356949 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:45:53.356901    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:46:00.620090 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:46:00.620034    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 18:46:01.234465 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:01.234430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:46:03.315652 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:03.315604    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:46:03.357361 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:03.357313    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:46:13.316321 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:13.316267    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:46:13.357515 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:13.357467    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:46:23.315879 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:23.315822    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:46:23.357956 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:23.357911    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:46:30.191981 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:30.191887    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:46:33.315470 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:33.315422    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:46:33.357062 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:33.357015    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:46:43.316360 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:43.316307    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:46:43.357908 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:43.357863    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:46:53.316180 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:53.316133    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:46:53.357115 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:53.357070    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:46:57.620631 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.620596    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"]
Jul 09 18:46:57.623097 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.623076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.625938 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.625910    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-m8pgz\""
Jul 09 18:46:57.625938 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.625929    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 09 18:46:57.627001 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.626977    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"]
Jul 09 18:46:57.628992 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.628976    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.644764 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.644736    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"]
Jul 09 18:46:57.648416 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.648383    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"]
Jul 09 18:46:57.669111 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.669074    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:46:57.718155 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.718113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f790da47-896e-405c-ba3f-608321ec51e5-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.718368 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.718172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn767\" (UniqueName: \"kubernetes.io/projected/f790da47-896e-405c-ba3f-608321ec51e5-kube-api-access-cn767\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.718368 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.718251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f790da47-896e-405c-ba3f-608321ec51e5-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.718368 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.718325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrzq\" (UniqueName: \"kubernetes.io/projected/e2fb5119-9261-41ca-8654-b7060a960639-kube-api-access-4jrzq\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.718368 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.718357    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f790da47-896e-405c-ba3f-608321ec51e5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.718527 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.718456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f790da47-896e-405c-ba3f-608321ec51e5-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.718563 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.718523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5119-9261-41ca-8654-b7060a960639-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.718597 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.718571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5119-9261-41ca-8654-b7060a960639-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.718677 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.718656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5119-9261-41ca-8654-b7060a960639-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.718714 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.718684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5119-9261-41ca-8654-b7060a960639-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.718747 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.718714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2fb5119-9261-41ca-8654-b7060a960639-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.718820 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.718775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f790da47-896e-405c-ba3f-608321ec51e5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.820086 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5119-9261-41ca-8654-b7060a960639-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.820322 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5119-9261-41ca-8654-b7060a960639-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.820322 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5119-9261-41ca-8654-b7060a960639-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.820322 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5119-9261-41ca-8654-b7060a960639-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.820322 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2fb5119-9261-41ca-8654-b7060a960639-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.820322 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f790da47-896e-405c-ba3f-608321ec51e5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.820322 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f790da47-896e-405c-ba3f-608321ec51e5-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.820322 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn767\" (UniqueName: \"kubernetes.io/projected/f790da47-896e-405c-ba3f-608321ec51e5-kube-api-access-cn767\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.820694 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f790da47-896e-405c-ba3f-608321ec51e5-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.820694 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrzq\" (UniqueName: \"kubernetes.io/projected/e2fb5119-9261-41ca-8654-b7060a960639-kube-api-access-4jrzq\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.820694 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f790da47-896e-405c-ba3f-608321ec51e5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.820694 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f790da47-896e-405c-ba3f-608321ec51e5-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.820974 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5119-9261-41ca-8654-b7060a960639-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.820974 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820882    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f790da47-896e-405c-ba3f-608321ec51e5-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.820974 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.820935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5119-9261-41ca-8654-b7060a960639-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.821141 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.821112    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f790da47-896e-405c-ba3f-608321ec51e5-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.821224 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.821202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f790da47-896e-405c-ba3f-608321ec51e5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.821432 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.821390    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5119-9261-41ca-8654-b7060a960639-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.822977 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.822886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f790da47-896e-405c-ba3f-608321ec51e5-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.823137 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.823113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5119-9261-41ca-8654-b7060a960639-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.823315 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.823298    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f790da47-896e-405c-ba3f-608321ec51e5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.823377 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.823337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2fb5119-9261-41ca-8654-b7060a960639-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:57.830302 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.830270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn767\" (UniqueName: \"kubernetes.io/projected/f790da47-896e-405c-ba3f-608321ec51e5-kube-api-access-cn767\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" (UID: \"f790da47-896e-405c-ba3f-608321ec51e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:46:57.830302 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:57.830292    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrzq\" (UniqueName: \"kubernetes.io/projected/e2fb5119-9261-41ca-8654-b7060a960639-kube-api-access-4jrzq\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" (UID: \"e2fb5119-9261-41ca-8654-b7060a960639\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:46:58.012139 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:46:58.012077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:47:03.315893 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:47:03.315851    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:47:03.357656 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:47:03.357614    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:47:13.316295 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:47:13.316250    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:47:13.357973 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:47:13.357925    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:47:23.316019 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:47:23.315963    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:47:23.357181 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:47:23.357139    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:47:24.337149 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:47:24.337109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:47:33.316380 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:47:33.316321    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:47:33.357863 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:47:33.357810    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:47:43.315561 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:47:43.315498    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:47:43.357728 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:47:43.357682    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:47:53.316076 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:47:53.316028    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:47:53.357180 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:47:53.357127    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:48:03.316377 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:48:03.316263    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Jul 09 18:48:03.357356 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:48:03.357314    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 09 18:48:04.235488 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:48:04.235441    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 18:48:04.824659 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:48:04.824622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:48:13.325711 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:48:13.325665    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:48:13.338194 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:48:13.338161    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:48:13.367670 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:48:13.367637    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:48:13.376571 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:48:13.376534    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:49:00.634632 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:49:00.634577    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 18:49:00.641942 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:49:00.641901    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 18:49:01.075052 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:01.075016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:49:01.075254 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:01.075017    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:49:03.237875 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:03.237832    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"]
Jul 09 18:49:03.240067 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:03.240042    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"]
Jul 09 18:49:03.270834 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:03.270773    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:49:04.452591 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:04.452560    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:49:18.262996 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:18.262951    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main" containerID="cri-o://8f4ddd29a1f25b7326d4cc543767752d6378c3576a3dafa943ec4582e87f30eb" gracePeriod=45
Jul 09 18:49:18.263399 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:18.263038    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main" containerID="cri-o://73bfa8d06b5f844fcbaeec5fa3a3991b8d934356ca2f20db6c724d872b424a94" gracePeriod=45
Jul 09 18:49:19.081601 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.081567    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="llm-d-routing-sidecar" containerID="cri-o://68a05ad9d12613441a68018ff7566d5c0fd00ed1aa07299c8f1477cfbd4fb31e" gracePeriod=45
Jul 09 18:49:19.158622 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.158589    2578 generic.go:356] "Generic (PLEG): container finished" podID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerID="8f4ddd29a1f25b7326d4cc543767752d6378c3576a3dafa943ec4582e87f30eb" exitCode=0
Jul 09 18:49:19.158869 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.158663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" event={"ID":"d64b5b28-6f43-43f6-b09b-64b20519a92a","Type":"ContainerDied","Data":"8f4ddd29a1f25b7326d4cc543767752d6378c3576a3dafa943ec4582e87f30eb"}
Jul 09 18:49:19.161376 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.161339    2578 generic.go:356] "Generic (PLEG): container finished" podID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerID="73bfa8d06b5f844fcbaeec5fa3a3991b8d934356ca2f20db6c724d872b424a94" exitCode=0
Jul 09 18:49:19.161376 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.161387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" event={"ID":"cd420cb5-4e2e-4536-979f-47fdd89b47e4","Type":"ContainerDied","Data":"73bfa8d06b5f844fcbaeec5fa3a3991b8d934356ca2f20db6c724d872b424a94"}
Jul 09 18:49:19.222054 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.222027    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:49:19.283304 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.283263    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64b5b28-6f43-43f6-b09b-64b20519a92a-tls-certs\") pod \"d64b5b28-6f43-43f6-b09b-64b20519a92a\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") "
Jul 09 18:49:19.283886 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.283334    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-kserve-provision-location\") pod \"d64b5b28-6f43-43f6-b09b-64b20519a92a\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") "
Jul 09 18:49:19.283886 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.283361    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-tmp-dir\") pod \"d64b5b28-6f43-43f6-b09b-64b20519a92a\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") "
Jul 09 18:49:19.283886 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.283407    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzsm8\" (UniqueName: \"kubernetes.io/projected/d64b5b28-6f43-43f6-b09b-64b20519a92a-kube-api-access-vzsm8\") pod \"d64b5b28-6f43-43f6-b09b-64b20519a92a\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") "
Jul 09 18:49:19.283886 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.283429    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-dshm\") pod \"d64b5b28-6f43-43f6-b09b-64b20519a92a\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") "
Jul 09 18:49:19.283886 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.283485    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-model-cache\") pod \"d64b5b28-6f43-43f6-b09b-64b20519a92a\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") "
Jul 09 18:49:19.283886 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.283507    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-home\") pod \"d64b5b28-6f43-43f6-b09b-64b20519a92a\" (UID: \"d64b5b28-6f43-43f6-b09b-64b20519a92a\") "
Jul 09 18:49:19.283886 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.283776    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d64b5b28-6f43-43f6-b09b-64b20519a92a" (UID: "d64b5b28-6f43-43f6-b09b-64b20519a92a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:49:19.284294 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.283911    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:49:19.284509 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.284482    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-home" (OuterVolumeSpecName: "home") pod "d64b5b28-6f43-43f6-b09b-64b20519a92a" (UID: "d64b5b28-6f43-43f6-b09b-64b20519a92a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:49:19.286103 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.285962    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64b5b28-6f43-43f6-b09b-64b20519a92a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d64b5b28-6f43-43f6-b09b-64b20519a92a" (UID: "d64b5b28-6f43-43f6-b09b-64b20519a92a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:49:19.286244 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.286190    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64b5b28-6f43-43f6-b09b-64b20519a92a-kube-api-access-vzsm8" (OuterVolumeSpecName: "kube-api-access-vzsm8") pod "d64b5b28-6f43-43f6-b09b-64b20519a92a" (UID: "d64b5b28-6f43-43f6-b09b-64b20519a92a"). InnerVolumeSpecName "kube-api-access-vzsm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:49:19.286396 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.286370    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-dshm" (OuterVolumeSpecName: "dshm") pod "d64b5b28-6f43-43f6-b09b-64b20519a92a" (UID: "d64b5b28-6f43-43f6-b09b-64b20519a92a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:49:19.298298 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.298246    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d64b5b28-6f43-43f6-b09b-64b20519a92a" (UID: "d64b5b28-6f43-43f6-b09b-64b20519a92a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:49:19.336282 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.336247    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:49:19.353646 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.353594    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d64b5b28-6f43-43f6-b09b-64b20519a92a" (UID: "d64b5b28-6f43-43f6-b09b-64b20519a92a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:49:19.384513 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.384474    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-kserve-provision-location\") pod \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") "
Jul 09 18:49:19.384513 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.384515    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klpf9\" (UniqueName: \"kubernetes.io/projected/cd420cb5-4e2e-4536-979f-47fdd89b47e4-kube-api-access-klpf9\") pod \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") "
Jul 09 18:49:19.384785 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.384606    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-tmp-dir\") pod \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") "
Jul 09 18:49:19.384785 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.384624    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-home\") pod \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") "
Jul 09 18:49:19.384785 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.384646    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-dshm\") pod \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") "
Jul 09 18:49:19.384785 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.384666    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-model-cache\") pod \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") "
Jul 09 18:49:19.384785 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.384706    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd420cb5-4e2e-4536-979f-47fdd89b47e4-tls-certs\") pod \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\" (UID: \"cd420cb5-4e2e-4536-979f-47fdd89b47e4\") "
Jul 09 18:49:19.385101 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.385062    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64b5b28-6f43-43f6-b09b-64b20519a92a-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:49:19.385101 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.385092    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:49:19.385207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.385106    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:49:19.385207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.385118    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzsm8\" (UniqueName: \"kubernetes.io/projected/d64b5b28-6f43-43f6-b09b-64b20519a92a-kube-api-access-vzsm8\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:49:19.385207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.385130    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:49:19.385207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.385142    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64b5b28-6f43-43f6-b09b-64b20519a92a-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:49:19.385207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.385156    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd420cb5-4e2e-4536-979f-47fdd89b47e4" (UID: "cd420cb5-4e2e-4536-979f-47fdd89b47e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:49:19.385475 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.385380    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-home" (OuterVolumeSpecName: "home") pod "cd420cb5-4e2e-4536-979f-47fdd89b47e4" (UID: "cd420cb5-4e2e-4536-979f-47fdd89b47e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:49:19.387555 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.387518    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-dshm" (OuterVolumeSpecName: "dshm") pod "cd420cb5-4e2e-4536-979f-47fdd89b47e4" (UID: "cd420cb5-4e2e-4536-979f-47fdd89b47e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:49:19.387555 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.387530    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd420cb5-4e2e-4536-979f-47fdd89b47e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd420cb5-4e2e-4536-979f-47fdd89b47e4" (UID: "cd420cb5-4e2e-4536-979f-47fdd89b47e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:49:19.387783 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.387527    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd420cb5-4e2e-4536-979f-47fdd89b47e4-kube-api-access-klpf9" (OuterVolumeSpecName: "kube-api-access-klpf9") pod "cd420cb5-4e2e-4536-979f-47fdd89b47e4" (UID: "cd420cb5-4e2e-4536-979f-47fdd89b47e4"). InnerVolumeSpecName "kube-api-access-klpf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:49:19.397780 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.397720    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd420cb5-4e2e-4536-979f-47fdd89b47e4" (UID: "cd420cb5-4e2e-4536-979f-47fdd89b47e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:49:19.452900 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.452852    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd420cb5-4e2e-4536-979f-47fdd89b47e4" (UID: "cd420cb5-4e2e-4536-979f-47fdd89b47e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:49:19.486438 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.486401    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:49:19.486438 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.486431    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:49:19.486438 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.486440    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:49:19.486648 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.486447    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:49:19.486648 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.486456    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd420cb5-4e2e-4536-979f-47fdd89b47e4-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:49:19.486648 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.486464    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd420cb5-4e2e-4536-979f-47fdd89b47e4-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:49:19.486648 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:19.486473    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klpf9\" (UniqueName: \"kubernetes.io/projected/cd420cb5-4e2e-4536-979f-47fdd89b47e4-kube-api-access-klpf9\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:49:20.167289 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.167250    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"
Jul 09 18:49:20.167289 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.167271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx" event={"ID":"d64b5b28-6f43-43f6-b09b-64b20519a92a","Type":"ContainerDied","Data":"6ad31879b53d4063966da569242686a435c66aa4ef5a61cd03bafab3fd90752d"}
Jul 09 18:49:20.167569 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.167343    2578 scope.go:117] "RemoveContainer" containerID="8f4ddd29a1f25b7326d4cc543767752d6378c3576a3dafa943ec4582e87f30eb"
Jul 09 18:49:20.169474 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.169443    2578 generic.go:356] "Generic (PLEG): container finished" podID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerID="68a05ad9d12613441a68018ff7566d5c0fd00ed1aa07299c8f1477cfbd4fb31e" exitCode=0
Jul 09 18:49:20.169618 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.169515    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"
Jul 09 18:49:20.169688 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.169513    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" event={"ID":"cd420cb5-4e2e-4536-979f-47fdd89b47e4","Type":"ContainerDied","Data":"68a05ad9d12613441a68018ff7566d5c0fd00ed1aa07299c8f1477cfbd4fb31e"}
Jul 09 18:49:20.169740 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.169684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff" event={"ID":"cd420cb5-4e2e-4536-979f-47fdd89b47e4","Type":"ContainerDied","Data":"305c66666fd233e70e6333620c40f803e65da9d0876842df8e0185a0dfc040e8"}
Jul 09 18:49:20.177323 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.177304    2578 scope.go:117] "RemoveContainer" containerID="40358f6d7d5639be55e398253813331f65bb55bed448879ad52ac6bf3c54406d"
Jul 09 18:49:20.189074 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.189044    2578 scope.go:117] "RemoveContainer" containerID="73bfa8d06b5f844fcbaeec5fa3a3991b8d934356ca2f20db6c724d872b424a94"
Jul 09 18:49:20.197937 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.197908    2578 scope.go:117] "RemoveContainer" containerID="db0e7e04283aaf52bb75db3418240ed7aa0f2e8aee99e9ac5f837d615e42af0d"
Jul 09 18:49:20.207122 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.207096    2578 scope.go:117] "RemoveContainer" containerID="68a05ad9d12613441a68018ff7566d5c0fd00ed1aa07299c8f1477cfbd4fb31e"
Jul 09 18:49:20.208665 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.208641    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"]
Jul 09 18:49:20.216941 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.216915    2578 scope.go:117] "RemoveContainer" containerID="73bfa8d06b5f844fcbaeec5fa3a3991b8d934356ca2f20db6c724d872b424a94"
Jul 09 18:49:20.217500 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.217469    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx"]
Jul 09 18:49:20.217664 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:49:20.217629    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73bfa8d06b5f844fcbaeec5fa3a3991b8d934356ca2f20db6c724d872b424a94\": container with ID starting with 73bfa8d06b5f844fcbaeec5fa3a3991b8d934356ca2f20db6c724d872b424a94 not found: ID does not exist" containerID="73bfa8d06b5f844fcbaeec5fa3a3991b8d934356ca2f20db6c724d872b424a94"
Jul 09 18:49:20.217726 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.217677    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73bfa8d06b5f844fcbaeec5fa3a3991b8d934356ca2f20db6c724d872b424a94"} err="failed to get container status \"73bfa8d06b5f844fcbaeec5fa3a3991b8d934356ca2f20db6c724d872b424a94\": rpc error: code = NotFound desc = could not find container \"73bfa8d06b5f844fcbaeec5fa3a3991b8d934356ca2f20db6c724d872b424a94\": container with ID starting with 73bfa8d06b5f844fcbaeec5fa3a3991b8d934356ca2f20db6c724d872b424a94 not found: ID does not exist"
Jul 09 18:49:20.217726 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.217706    2578 scope.go:117] "RemoveContainer" containerID="db0e7e04283aaf52bb75db3418240ed7aa0f2e8aee99e9ac5f837d615e42af0d"
Jul 09 18:49:20.221339 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:49:20.221300    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0e7e04283aaf52bb75db3418240ed7aa0f2e8aee99e9ac5f837d615e42af0d\": container with ID starting with db0e7e04283aaf52bb75db3418240ed7aa0f2e8aee99e9ac5f837d615e42af0d not found: ID does not exist" containerID="db0e7e04283aaf52bb75db3418240ed7aa0f2e8aee99e9ac5f837d615e42af0d"
Jul 09 18:49:20.221500 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.221351    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0e7e04283aaf52bb75db3418240ed7aa0f2e8aee99e9ac5f837d615e42af0d"} err="failed to get container status \"db0e7e04283aaf52bb75db3418240ed7aa0f2e8aee99e9ac5f837d615e42af0d\": rpc error: code = NotFound desc = could not find container \"db0e7e04283aaf52bb75db3418240ed7aa0f2e8aee99e9ac5f837d615e42af0d\": container with ID starting with db0e7e04283aaf52bb75db3418240ed7aa0f2e8aee99e9ac5f837d615e42af0d not found: ID does not exist"
Jul 09 18:49:20.221500 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.221379    2578 scope.go:117] "RemoveContainer" containerID="68a05ad9d12613441a68018ff7566d5c0fd00ed1aa07299c8f1477cfbd4fb31e"
Jul 09 18:49:20.223867 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:49:20.223832    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a05ad9d12613441a68018ff7566d5c0fd00ed1aa07299c8f1477cfbd4fb31e\": container with ID starting with 68a05ad9d12613441a68018ff7566d5c0fd00ed1aa07299c8f1477cfbd4fb31e not found: ID does not exist" containerID="68a05ad9d12613441a68018ff7566d5c0fd00ed1aa07299c8f1477cfbd4fb31e"
Jul 09 18:49:20.224011 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.223877    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a05ad9d12613441a68018ff7566d5c0fd00ed1aa07299c8f1477cfbd4fb31e"} err="failed to get container status \"68a05ad9d12613441a68018ff7566d5c0fd00ed1aa07299c8f1477cfbd4fb31e\": rpc error: code = NotFound desc = could not find container \"68a05ad9d12613441a68018ff7566d5c0fd00ed1aa07299c8f1477cfbd4fb31e\": container with ID starting with 68a05ad9d12613441a68018ff7566d5c0fd00ed1aa07299c8f1477cfbd4fb31e not found: ID does not exist"
Jul 09 18:49:20.232068 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.232029    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"]
Jul 09 18:49:20.236094 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:20.236056    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff"]
Jul 09 18:49:21.403498 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:21.403459    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" path="/var/lib/kubelet/pods/cd420cb5-4e2e-4536-979f-47fdd89b47e4/volumes"
Jul 09 18:49:21.404031 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:21.404014    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" path="/var/lib/kubelet/pods/d64b5b28-6f43-43f6-b09b-64b20519a92a/volumes"
Jul 09 18:49:37.460111 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:37.460033    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:49:37.468403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:37.468380    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:49:37.468580 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:37.468427    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:49:37.476651 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:37.476625    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:49:52.772075 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.772041    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"]
Jul 09 18:49:52.772497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.772386    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="llm-d-routing-sidecar"
Jul 09 18:49:52.772497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.772398    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="llm-d-routing-sidecar"
Jul 09 18:49:52.772497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.772406    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="storage-initializer"
Jul 09 18:49:52.772497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.772412    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="storage-initializer"
Jul 09 18:49:52.772497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.772420    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main"
Jul 09 18:49:52.772497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.772426    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main"
Jul 09 18:49:52.772497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.772434    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="storage-initializer"
Jul 09 18:49:52.772497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.772439    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="storage-initializer"
Jul 09 18:49:52.772497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.772459    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main"
Jul 09 18:49:52.772497 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.772465    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main"
Jul 09 18:49:52.772858 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.772519    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="main"
Jul 09 18:49:52.772858 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.772529    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d64b5b28-6f43-43f6-b09b-64b20519a92a" containerName="main"
Jul 09 18:49:52.772858 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.772537    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd420cb5-4e2e-4536-979f-47fdd89b47e4" containerName="llm-d-routing-sidecar"
Jul 09 18:49:52.775760 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.775741    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.778644 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.778614    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 09 18:49:52.778781 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.778620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:49:52.788961 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.788930    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"]
Jul 09 18:49:52.795557 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.795528    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:49:52.802861 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.802830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.803030 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.802895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.803030 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.802989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.803030 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.803010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.803030 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.803029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.803205 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.803076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.803205 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.803161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpmzf\" (UniqueName: \"kubernetes.io/projected/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-kube-api-access-fpmzf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.904184 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.904141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.904374 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.904197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.904374 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.904236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.904374 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.904254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.904374 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.904271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.904374 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.904297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.904374 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.904335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpmzf\" (UniqueName: \"kubernetes.io/projected/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-kube-api-access-fpmzf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.904698 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.904642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.904698 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.904652    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.904838 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.904708    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.904838 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.904735    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.906708 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.906675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.906949 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.906929    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:52.914563 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:52.914532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpmzf\" (UniqueName: \"kubernetes.io/projected/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-kube-api-access-fpmzf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:53.085871 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:53.085770    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:49:53.234938 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:53.234908    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"]
Jul 09 18:49:53.236924 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:49:53.236891    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3c4061_cbfa_4fb6_91cf_009988a7ffe4.slice/crio-58b111aa448cd4c89d7cace352fdee1a4a23d66148cd8ba3d2df9d8eb0137e7b WatchSource:0}: Error finding container 58b111aa448cd4c89d7cace352fdee1a4a23d66148cd8ba3d2df9d8eb0137e7b: Status 404 returned error can't find the container with id 58b111aa448cd4c89d7cace352fdee1a4a23d66148cd8ba3d2df9d8eb0137e7b
Jul 09 18:49:53.313648 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:53.313616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" event={"ID":"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4","Type":"ContainerStarted","Data":"ebfb6854b094e6c1d4338abc843b37690dedb09c026557d4d22e3172ed3c06ca"}
Jul 09 18:49:53.313771 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:53.313670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" event={"ID":"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4","Type":"ContainerStarted","Data":"58b111aa448cd4c89d7cace352fdee1a4a23d66148cd8ba3d2df9d8eb0137e7b"}
Jul 09 18:49:57.336703 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:57.336660    2578 generic.go:356] "Generic (PLEG): container finished" podID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerID="ebfb6854b094e6c1d4338abc843b37690dedb09c026557d4d22e3172ed3c06ca" exitCode=0
Jul 09 18:49:57.337124 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:57.336735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" event={"ID":"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4","Type":"ContainerDied","Data":"ebfb6854b094e6c1d4338abc843b37690dedb09c026557d4d22e3172ed3c06ca"}
Jul 09 18:49:58.343093 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:58.343052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" event={"ID":"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4","Type":"ContainerStarted","Data":"ba02b149d5c000d4b7bd8306805deb9fc42f0181efec73171c805e47686fb7a9"}
Jul 09 18:49:58.370217 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:58.370156    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podStartSLOduration=2.850115938 podStartE2EDuration="6.370138566s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.520022628s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:49:58.366894597 +0000 UTC m=+2721.553872383" watchObservedRunningTime="2026-07-09 18:49:58.370138566 +0000 UTC m=+2721.557116334"
Jul 09 18:49:59.086409 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:49:59.086374    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:50:03.086636 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:50:03.086591    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:50:03.089014 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:50:03.088445    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 09 18:50:07.826152 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:50:07.826102    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 18:50:08.398298 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:50:08.398268    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:50:13.086233 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:50:13.086180    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 09 18:50:22.523547 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:50:22.523510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:50:22.596647 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:50:22.596610    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:50:23.087241 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:50:23.087201    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 09 18:50:33.086590 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:50:33.086531    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 09 18:50:43.086250 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:50:43.086203    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 09 18:50:53.086982 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:50:53.086928    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 09 18:50:59.931540 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:50:59.931451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:51:03.087264 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:51:03.087210    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 09 18:51:04.075678 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:51:04.075631    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 18:51:04.075893 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:51:04.075700    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 18:51:04.654298 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:51:04.654269    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:51:04.654298 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:51:04.654290    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:51:13.086543 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:51:13.086492    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 09 18:51:23.086244 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:51:23.086199    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 09 18:51:33.086408 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:51:33.086360    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 09 18:51:43.086451 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:51:43.086397    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 09 18:51:43.100063 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:51:43.100029    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:51:53.086609 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:51:53.086562    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 09 18:52:03.096786 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:03.096746    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:52:03.104884 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:03.104857    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:52:11.399196 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:52:11.399144    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 18:52:11.983461 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:11.983430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:52:38.424098 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:38.424016    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"]
Jul 09 18:52:38.440548 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:38.440513    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:52:52.970373 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:52.970332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:52:53.448877 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:53.448807    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main" containerID="cri-o://ba02b149d5c000d4b7bd8306805deb9fc42f0181efec73171c805e47686fb7a9" gracePeriod=45
Jul 09 18:52:54.098212 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.098171    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 09 18:52:54.169407 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.169374    2578 generic.go:356] "Generic (PLEG): container finished" podID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerID="ba02b149d5c000d4b7bd8306805deb9fc42f0181efec73171c805e47686fb7a9" exitCode=0
Jul 09 18:52:54.169581 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.169435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" event={"ID":"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4","Type":"ContainerDied","Data":"ba02b149d5c000d4b7bd8306805deb9fc42f0181efec73171c805e47686fb7a9"}
Jul 09 18:52:54.203161 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.203134    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:52:54.353156 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.353062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-model-cache\") pod \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") "
Jul 09 18:52:54.353156 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.353109    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-tmp-dir\") pod \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") "
Jul 09 18:52:54.353384 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.353161    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-kserve-provision-location\") pod \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") "
Jul 09 18:52:54.353384 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.353189    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpmzf\" (UniqueName: \"kubernetes.io/projected/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-kube-api-access-fpmzf\") pod \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") "
Jul 09 18:52:54.353384 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.353253    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-tls-certs\") pod \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") "
Jul 09 18:52:54.353384 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.353298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-home\") pod \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") "
Jul 09 18:52:54.353384 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.353337    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-dshm\") pod \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\" (UID: \"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4\") "
Jul 09 18:52:54.353642 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.353403    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" (UID: "4b3c4061-cbfa-4fb6-91cf-009988a7ffe4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:52:54.353705 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.353668    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:52:54.354065 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.354029    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-home" (OuterVolumeSpecName: "home") pod "4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" (UID: "4b3c4061-cbfa-4fb6-91cf-009988a7ffe4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:52:54.355637 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.355601    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-kube-api-access-fpmzf" (OuterVolumeSpecName: "kube-api-access-fpmzf") pod "4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" (UID: "4b3c4061-cbfa-4fb6-91cf-009988a7ffe4"). InnerVolumeSpecName "kube-api-access-fpmzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:52:54.355747 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.355703    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-dshm" (OuterVolumeSpecName: "dshm") pod "4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" (UID: "4b3c4061-cbfa-4fb6-91cf-009988a7ffe4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:52:54.355837 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.355817    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" (UID: "4b3c4061-cbfa-4fb6-91cf-009988a7ffe4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:52:54.372095 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.372059    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" (UID: "4b3c4061-cbfa-4fb6-91cf-009988a7ffe4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:52:54.423814 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.423736    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" (UID: "4b3c4061-cbfa-4fb6-91cf-009988a7ffe4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:52:54.454207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.454173    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:52:54.454207 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.454205    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:52:54.454364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.454217    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpmzf\" (UniqueName: \"kubernetes.io/projected/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-kube-api-access-fpmzf\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:52:54.454364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.454226    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:52:54.454364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.454234    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:52:54.454364 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:54.454242    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 18:52:55.175897 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:55.175851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222" event={"ID":"4b3c4061-cbfa-4fb6-91cf-009988a7ffe4","Type":"ContainerDied","Data":"58b111aa448cd4c89d7cace352fdee1a4a23d66148cd8ba3d2df9d8eb0137e7b"}
Jul 09 18:52:55.175897 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:55.175881    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"
Jul 09 18:52:55.176440 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:55.175954    2578 scope.go:117] "RemoveContainer" containerID="ba02b149d5c000d4b7bd8306805deb9fc42f0181efec73171c805e47686fb7a9"
Jul 09 18:52:55.185413 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:55.185393    2578 scope.go:117] "RemoveContainer" containerID="ebfb6854b094e6c1d4338abc843b37690dedb09c026557d4d22e3172ed3c06ca"
Jul 09 18:52:55.201020 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:55.200991    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"]
Jul 09 18:52:55.202566 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:55.202537    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222"]
Jul 09 18:52:55.403231 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:52:55.403195    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" path="/var/lib/kubelet/pods/4b3c4061-cbfa-4fb6-91cf-009988a7ffe4/volumes"
Jul 09 18:53:07.655402 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:53:07.655352    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 18:53:07.655402 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:53:07.655366    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 18:53:08.232392 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:08.232356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:53:08.232575 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:08.232356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:53:34.034651 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.034614    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:34.253450 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.253413    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:34.380085 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.379991    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:53:34.380420 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.380402    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="storage-initializer"
Jul 09 18:53:34.380506 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.380423    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="storage-initializer"
Jul 09 18:53:34.380506 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.380436    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main"
Jul 09 18:53:34.380506 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.380444    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main"
Jul 09 18:53:34.380663 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.380541    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b3c4061-cbfa-4fb6-91cf-009988a7ffe4" containerName="main"
Jul 09 18:53:34.383379 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.383355    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.387853 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.387788    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:53:34.388013 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.387864    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 09 18:53:34.388153 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.388134    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-dr75p\""
Jul 09 18:53:34.397384 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.397350    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:53:34.512751 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.512713    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:34.534814 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.534746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.535001 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.534833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.535001 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.534887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.535117 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.535101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a8b35d-132c-4f40-834b-8e546707a7f4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.535173 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.535158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h689f\" (UniqueName: \"kubernetes.io/projected/99a8b35d-132c-4f40-834b-8e546707a7f4-kube-api-access-h689f\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.535227 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.535200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.535274 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.535263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.626403 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.626369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:34.636775 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.636685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a8b35d-132c-4f40-834b-8e546707a7f4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.636775 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.636744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h689f\" (UniqueName: \"kubernetes.io/projected/99a8b35d-132c-4f40-834b-8e546707a7f4-kube-api-access-h689f\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.637046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.636782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.637046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.636859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.637046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.636898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.637046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.636930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.637046 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.636968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.637354 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.637328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.637428 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.637375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.637428 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.637397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.637562 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.637546    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.639246 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.639218    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.639365 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.639313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a8b35d-132c-4f40-834b-8e546707a7f4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.656523 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.656489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h689f\" (UniqueName: \"kubernetes.io/projected/99a8b35d-132c-4f40-834b-8e546707a7f4-kube-api-access-h689f\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.694634 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.694593    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:34.842415 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:34.842287    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:53:34.845216 ip-10-0-137-79 kubenswrapper[2578]: W0709 18:53:34.845190    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a8b35d_132c_4f40_834b_8e546707a7f4.slice/crio-12264220678a07029cc0fd53538cd4b8b931de770bcab191e713a0c8f55a450f WatchSource:0}: Error finding container 12264220678a07029cc0fd53538cd4b8b931de770bcab191e713a0c8f55a450f: Status 404 returned error can't find the container with id 12264220678a07029cc0fd53538cd4b8b931de770bcab191e713a0c8f55a450f
Jul 09 18:53:35.353816 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:35.353756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"99a8b35d-132c-4f40-834b-8e546707a7f4","Type":"ContainerStarted","Data":"603f0ef87f235504bb5f30e43252ecda2012e6f16f9614c83fc52761105756c2"}
Jul 09 18:53:35.354315 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:35.353845    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"99a8b35d-132c-4f40-834b-8e546707a7f4","Type":"ContainerStarted","Data":"12264220678a07029cc0fd53538cd4b8b931de770bcab191e713a0c8f55a450f"}
Jul 09 18:53:39.373759 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:39.373722    2578 generic.go:356] "Generic (PLEG): container finished" podID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerID="603f0ef87f235504bb5f30e43252ecda2012e6f16f9614c83fc52761105756c2" exitCode=0
Jul 09 18:53:39.374175 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:39.373784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"99a8b35d-132c-4f40-834b-8e546707a7f4","Type":"ContainerDied","Data":"603f0ef87f235504bb5f30e43252ecda2012e6f16f9614c83fc52761105756c2"}
Jul 09 18:53:40.379698 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:40.379657    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"99a8b35d-132c-4f40-834b-8e546707a7f4","Type":"ContainerStarted","Data":"4aa98c8dbe960036f9116b2f08fe16684a2cf32f90bd19ef3621b4f3b0a894db"}
Jul 09 18:53:40.404076 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:40.404022    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.596776284 podStartE2EDuration="6.40400507s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.807228786s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:40.402917296 +0000 UTC m=+2943.589895087" watchObservedRunningTime="2026-07-09 18:53:40.40400507 +0000 UTC m=+2943.590982892"
Jul 09 18:53:40.695769 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:40.695717    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:44.694984 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:44.694940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:44.696453 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:44.696413    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 09 18:53:54.695294 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:53:54.695237    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 09 18:54:04.695666 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:54:04.695571    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 09 18:54:14.695665 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:54:14.695614    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 09 18:54:14.984898 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:54:14.984847    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 18:54:15.541847 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:54:15.541788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:54:18.392043 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:54:18.392000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:24.695356 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:54:24.695305    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 09 18:54:34.695824 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:54:34.695742    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 09 18:54:37.488393 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:54:37.488357    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:54:37.497474 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:54:37.497445    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:54:37.497956 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:54:37.497915    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:54:37.505997 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:54:37.505973    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:54:44.696005 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:54:44.695957    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 09 18:54:54.696413 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:54:54.696372    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 09 18:55:04.695945 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:55:04.695897    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 09 18:55:11.234028 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:55:11.233976    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 18:55:11.234028 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:55:11.234021    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 18:55:11.814811 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:55:11.814757    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:55:11.814811 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:55:11.814776    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:55:14.695871 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:55:14.695787    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 09 18:55:24.705140 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:55:24.705109    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:55:24.713009 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:55:24.712972    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:55:45.605676 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:55:45.605561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:49.578518 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:55:49.578482    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:18.542762 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:56:18.542709    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 18:56:19.111758 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:56:19.111726    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:57:14.816033 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:57:14.815929    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 18:57:14.816033 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:57:14.815962    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 18:57:15.367674 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:57:15.367638    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:57:15.367882 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:57:15.367638    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:58:22.112544 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:58:22.112498    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 18:58:22.664992 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:58:22.664958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 18:59:18.371030 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:59:18.370975    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 18:59:18.372816 ip-10-0-137-79 kubenswrapper[2578]: E0709 18:59:18.372746    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 18:59:18.931107 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:59:18.931074    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 18:59:18.931297 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:59:18.931088    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 18:59:37.516679 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:59:37.516648    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:59:37.525477 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:59:37.525448    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 18:59:37.526417 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:59:37.526391    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 18:59:37.534448 ip-10-0-137-79 kubenswrapper[2578]: I0709 18:59:37.534424    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 19:00:25.666591 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:00:25.666543    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:00:26.239307 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:00:26.239274    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:01:21.195135 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:01:21.195097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:21.932442 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:01:21.932392    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 19:01:21.932442 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:01:21.932392    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 19:01:22.487901 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:01:22.487869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 19:01:22.488296 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:01:22.487872    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 19:01:25.153721 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:01:25.153686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:41.099820 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:01:41.099705    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:05.692858 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:05.692818    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:05.848304 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:05.848268    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 19:02:05.851701 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:05.851682    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:05.860286 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:05.860258    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-k5ff9\""
Jul 09 19:02:05.861443 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:05.861420    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 09 19:02:05.882823 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:05.882770    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 19:02:05.924875 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:05.924840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:05.925096 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:05.924966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:05.925096 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:05.925007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:05.925096 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:05.925030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:05.925096 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:05.925052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87l4n\" (UniqueName: \"kubernetes.io/projected/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-kube-api-access-87l4n\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:05.925096 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:05.925076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:05.969226 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:05.969189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:06.025888 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:06.025854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:06.026100 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:06.025985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:06.026100 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:06.026025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:06.026100 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:06.026050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:06.026100 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:06.026073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87l4n\" (UniqueName: \"kubernetes.io/projected/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-kube-api-access-87l4n\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:06.026100 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:06.026099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:06.026501 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:06.026476    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:06.026653 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:06.026562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:06.026765 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:06.026586    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:06.028355 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:06.028330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:06.028750 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:06.028733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:06.043857 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:06.043829    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87l4n\" (UniqueName: \"kubernetes.io/projected/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-kube-api-access-87l4n\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:02:06.118142 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:06.118092    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:29.240513 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:02:29.240464    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:02:29.785411 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:29.785380    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:02:42.781576 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:02:42.781536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:03:25.489062 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:03:25.489020    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 19:03:25.489062 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:03:25.489039    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 19:03:25.510354 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:25.510309    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:03:25.526082 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:25.526049    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 19:03:25.610214 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:25.610156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:03:26.037676 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:26.037642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 19:03:26.037876 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:26.037648    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 19:03:40.551012 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:40.550970    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main" containerID="cri-o://4aa98c8dbe960036f9116b2f08fe16684a2cf32f90bd19ef3621b4f3b0a894db" gracePeriod=45
Jul 09 19:03:40.705833 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:40.705775    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 09 19:03:41.103104 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.103066    2578 generic.go:356] "Generic (PLEG): container finished" podID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerID="4aa98c8dbe960036f9116b2f08fe16684a2cf32f90bd19ef3621b4f3b0a894db" exitCode=0
Jul 09 19:03:41.103277 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.103135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"99a8b35d-132c-4f40-834b-8e546707a7f4","Type":"ContainerDied","Data":"4aa98c8dbe960036f9116b2f08fe16684a2cf32f90bd19ef3621b4f3b0a894db"}
Jul 09 19:03:41.233339 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.233313    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 19:03:41.374354 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.374245    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-kserve-provision-location\") pod \"99a8b35d-132c-4f40-834b-8e546707a7f4\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") "
Jul 09 19:03:41.374354 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.374313    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-model-cache\") pod \"99a8b35d-132c-4f40-834b-8e546707a7f4\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") "
Jul 09 19:03:41.374354 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.374342    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a8b35d-132c-4f40-834b-8e546707a7f4-tls-certs\") pod \"99a8b35d-132c-4f40-834b-8e546707a7f4\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") "
Jul 09 19:03:41.374627 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.374384    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-home\") pod \"99a8b35d-132c-4f40-834b-8e546707a7f4\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") "
Jul 09 19:03:41.374627 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.374435    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-dshm\") pod \"99a8b35d-132c-4f40-834b-8e546707a7f4\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") "
Jul 09 19:03:41.374627 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.374501    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-tmp-dir\") pod \"99a8b35d-132c-4f40-834b-8e546707a7f4\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") "
Jul 09 19:03:41.374627 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.374533    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h689f\" (UniqueName: \"kubernetes.io/projected/99a8b35d-132c-4f40-834b-8e546707a7f4-kube-api-access-h689f\") pod \"99a8b35d-132c-4f40-834b-8e546707a7f4\" (UID: \"99a8b35d-132c-4f40-834b-8e546707a7f4\") "
Jul 09 19:03:41.374627 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.374561    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "99a8b35d-132c-4f40-834b-8e546707a7f4" (UID: "99a8b35d-132c-4f40-834b-8e546707a7f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:41.374951 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.374927    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 19:03:41.375151 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.375124    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-home" (OuterVolumeSpecName: "home") pod "99a8b35d-132c-4f40-834b-8e546707a7f4" (UID: "99a8b35d-132c-4f40-834b-8e546707a7f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:41.376712 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.376679    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a8b35d-132c-4f40-834b-8e546707a7f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99a8b35d-132c-4f40-834b-8e546707a7f4" (UID: "99a8b35d-132c-4f40-834b-8e546707a7f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:03:41.377034 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.377004    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a8b35d-132c-4f40-834b-8e546707a7f4-kube-api-access-h689f" (OuterVolumeSpecName: "kube-api-access-h689f") pod "99a8b35d-132c-4f40-834b-8e546707a7f4" (UID: "99a8b35d-132c-4f40-834b-8e546707a7f4"). InnerVolumeSpecName "kube-api-access-h689f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:03:41.377139 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.377032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-dshm" (OuterVolumeSpecName: "dshm") pod "99a8b35d-132c-4f40-834b-8e546707a7f4" (UID: "99a8b35d-132c-4f40-834b-8e546707a7f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:41.388130 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.388061    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99a8b35d-132c-4f40-834b-8e546707a7f4" (UID: "99a8b35d-132c-4f40-834b-8e546707a7f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:41.444471 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.444422    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99a8b35d-132c-4f40-834b-8e546707a7f4" (UID: "99a8b35d-132c-4f40-834b-8e546707a7f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:41.475930 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.475890    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a8b35d-132c-4f40-834b-8e546707a7f4-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 19:03:41.475930 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.475923    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 19:03:41.475930 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.475932    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 19:03:41.475930 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.475940    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 19:03:41.476181 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.475948    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h689f\" (UniqueName: \"kubernetes.io/projected/99a8b35d-132c-4f40-834b-8e546707a7f4-kube-api-access-h689f\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 19:03:41.476181 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:41.475959    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99a8b35d-132c-4f40-834b-8e546707a7f4-kserve-provision-location\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 19:03:42.109737 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:42.109698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"99a8b35d-132c-4f40-834b-8e546707a7f4","Type":"ContainerDied","Data":"12264220678a07029cc0fd53538cd4b8b931de770bcab191e713a0c8f55a450f"}
Jul 09 19:03:42.110181 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:42.109773    2578 scope.go:117] "RemoveContainer" containerID="4aa98c8dbe960036f9116b2f08fe16684a2cf32f90bd19ef3621b4f3b0a894db"
Jul 09 19:03:42.110181 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:42.109710    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 19:03:42.119493 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:42.119475    2578 scope.go:117] "RemoveContainer" containerID="603f0ef87f235504bb5f30e43252ecda2012e6f16f9614c83fc52761105756c2"
Jul 09 19:03:42.135643 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:42.135612    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 19:03:42.137182 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:42.137159    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 19:03:43.405254 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:03:43.405210    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" path="/var/lib/kubelet/pods/99a8b35d-132c-4f40-834b-8e546707a7f4/volumes"
Jul 09 19:04:08.862298 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:04:08.862252    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5fdfb8f5-d1c5-4996-b928-214cbcf730d7"
Jul 09 19:04:09.230199 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:04:09.230153    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:04:16.530552 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:04:16.530512    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:04:16.630944 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:04:16.630904    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:04:32.786181 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:04:32.786073    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:04:33.340376 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:04:33.340345    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:04:35.158149 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:04:35.158060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:04:37.543478 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:04:37.543449    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 19:04:37.551743 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:04:37.551716    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 19:04:37.553393 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:04:37.553371    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 19:04:37.560784 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:04:37.560767    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 19:04:46.225705 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:04:46.225669    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:04:46.513222 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:04:46.513119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:04:51.126087 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:04:51.126047    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:04:51.317176 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:04:51.317141    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:05:07.311494 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.311452    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 19:05:07.312004 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:05:07.311811    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context canceled" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5fdfb8f5-d1c5-4996-b928-214cbcf730d7"
Jul 09 19:05:07.478858 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.478815    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:05:07.485890 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.485858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:05:07.496105 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.496076    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-dshm\") pod \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") "
Jul 09 19:05:07.496228 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.496122    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-tls-certs\") pod \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") "
Jul 09 19:05:07.496228 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.496154    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-home\") pod \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") "
Jul 09 19:05:07.496228 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.496199    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-tmp-dir\") pod \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") "
Jul 09 19:05:07.496228 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.496220    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87l4n\" (UniqueName: \"kubernetes.io/projected/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-kube-api-access-87l4n\") pod \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") "
Jul 09 19:05:07.496432 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.496289    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-model-cache\") pod \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\" (UID: \"5fdfb8f5-d1c5-4996-b928-214cbcf730d7\") "
Jul 09 19:05:07.496508 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.496474    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5fdfb8f5-d1c5-4996-b928-214cbcf730d7" (UID: "5fdfb8f5-d1c5-4996-b928-214cbcf730d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:05:07.496653 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.496513    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-home" (OuterVolumeSpecName: "home") pod "5fdfb8f5-d1c5-4996-b928-214cbcf730d7" (UID: "5fdfb8f5-d1c5-4996-b928-214cbcf730d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:05:07.496653 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.496565    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "5fdfb8f5-d1c5-4996-b928-214cbcf730d7" (UID: "5fdfb8f5-d1c5-4996-b928-214cbcf730d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:05:07.497680 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.496706    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-home\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 19:05:07.497680 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.496726    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-tmp-dir\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 19:05:07.497680 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.496740    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-model-cache\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 19:05:07.498920 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.498888    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-kube-api-access-87l4n" (OuterVolumeSpecName: "kube-api-access-87l4n") pod "5fdfb8f5-d1c5-4996-b928-214cbcf730d7" (UID: "5fdfb8f5-d1c5-4996-b928-214cbcf730d7"). InnerVolumeSpecName "kube-api-access-87l4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:05:07.498920 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.498906    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-dshm" (OuterVolumeSpecName: "dshm") pod "5fdfb8f5-d1c5-4996-b928-214cbcf730d7" (UID: "5fdfb8f5-d1c5-4996-b928-214cbcf730d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:05:07.498920 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.498906    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fdfb8f5-d1c5-4996-b928-214cbcf730d7" (UID: "5fdfb8f5-d1c5-4996-b928-214cbcf730d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:05:07.597912 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.597828    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-tls-certs\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 19:05:07.597912 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.597857    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87l4n\" (UniqueName: \"kubernetes.io/projected/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-kube-api-access-87l4n\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 19:05:07.597912 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:07.597866    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fdfb8f5-d1c5-4996-b928-214cbcf730d7-dshm\") on node \"ip-10-0-137-79.ec2.internal\" DevicePath \"\""
Jul 09 19:05:08.482972 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:08.482939    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 19:05:08.523676 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:08.523641    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 19:05:08.525173 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:08.525150    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 19:05:09.402979 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:09.402941    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fdfb8f5-d1c5-4996-b928-214cbcf730d7" path="/var/lib/kubelet/pods/5fdfb8f5-d1c5-4996-b928-214cbcf730d7/volumes"
Jul 09 19:05:09.820601 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:09.820553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:05:29.039179 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:05:29.039131    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 19:05:29.039179 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:05:29.039142    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 19:05:29.568469 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:29.568440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 19:05:29.568663 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:05:29.568440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 19:06:03.518620 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:06:03.518514    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:06:04.024164 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:06:04.024118    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:06:28.644170 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:06:28.644123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:06:30.106007 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:06:30.105966    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:06:36.341763 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:06:36.341709    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:06:36.848652 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:06:36.848619    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:06:53.538236 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:06:53.538191    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:07:25.103726 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:07:25.103681    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:07:25.296443 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:07:25.296408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:07:32.569959 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:07:32.569842    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 19:07:32.569959 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:07:32.569844    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 19:07:33.081860 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:07:33.081827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 19:07:33.082046 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:07:33.081827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 19:08:06.964306 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:08:06.964263    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:08:07.063684 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:08:07.063644    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:08:39.849528 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:08:39.849481    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:08:40.354659 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:08:40.354627    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:08:51.502701 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:08:51.502661    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:08:51.571977 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:08:51.571943    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:08:52.060569 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:08:52.060533    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:08:53.366492 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:08:53.366462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:08:53.439449 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:08:53.439411    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:09:19.215069 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:09:19.215021    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:09:19.420297 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:09:19.420254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:09:36.082572 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:09:36.082514    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 19:09:36.082572 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:09:36.082546    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 19:09:36.588586 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:09:36.588542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 19:09:36.589090 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:09:36.589071    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 19:09:37.569145 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:09:37.569117    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 19:09:37.577391 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:09:37.577366    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 19:09:37.578596 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:09:37.578570    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 19:09:37.585928 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:09:37.585908    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 19:09:41.169262 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:09:41.169219    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:09:41.427226 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:09:41.427137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:10:16.394977 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:10:16.394933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:10:43.355339 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:10:43.355288    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:10:43.859961 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:10:43.859928    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:10:44.639196 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:10:44.639160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:10:56.444038 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:10:56.443989    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:10:56.546402 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:10:56.546363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:11:05.640023 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:11:05.639986    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:11:05.713109 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:11:05.713066    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:11:33.020863 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:11:33.020819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:11:39.590022 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:11:39.589976    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 19:11:39.590569 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:11:39.590036    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 19:11:40.091657 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:11:40.091618    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 19:11:40.091861 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:11:40.091618    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 19:12:22.721076 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:12:22.721033    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:12:22.946517 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:12:22.946477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:12:37.116176 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:12:37.116138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:12:37.188024 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:12:37.187987    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:12:46.861199 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:12:46.861133    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:12:47.371516 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:12:47.371487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:12:54.048847 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:12:54.048786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:13:33.388173 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:13:33.388079    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:13:43.092397 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:13:43.092355    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 19:13:43.092809 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:13:43.092360    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 19:13:43.604530 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:13:43.604498    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 19:13:43.604719 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:13:43.604498    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 19:14:00.750552 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:14:00.750511    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:14:00.826039 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:14:00.826002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:14:37.595254 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:14:37.595225    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 19:14:37.603537 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:14:37.603502    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 19:14:37.604181 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:14:37.604160    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 19:14:37.612260 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:14:37.612234    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 19:14:50.372459 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:14:50.372405    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:14:50.879323 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:14:50.879290    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:15:01.622255 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:15:01.622161    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:15:02.028112 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:15:02.028073    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:15:46.605500 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:15:46.605445    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 19:15:46.605500 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:15:46.605446    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 19:15:47.106612 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:15:47.106583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 19:15:47.106833 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:15:47.106583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 19:15:47.824091 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:15:47.824045    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:16:05.372960 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:16:05.372924    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:16:05.442729 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:16:05.442679    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:16:08.373512 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:16:08.373472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:16:19.576855 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:16:19.576819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:16:22.487592 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:16:22.487558    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:16:53.880902 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:16:53.880857    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:16:54.384053 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:16:54.384018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:17:03.892308 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:17:03.892261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:17:05.345621 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:17:05.345578    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:17:20.885370 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:17:20.885326    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:17:31.994251 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:17:31.994195    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:17:32.093139 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:17:32.093095    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:17:50.107924 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:17:50.107880    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 19:17:50.108335 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:17:50.108036    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 19:17:50.617224 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:17:50.617193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 19:17:50.617224 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:17:50.617234    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 19:18:01.673414 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:18:01.673318    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:41.130447 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:18:41.130406    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:41.201838 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:18:41.201789    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:57.385254 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:18:57.385203    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:18:57.888902 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:18:57.888859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:19:26.841557 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:19:26.841521    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:33.541743 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:19:33.541652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:34.014249 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:19:34.014211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:37.620398 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:19:37.620372    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 19:19:37.628906 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:19:37.628874    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 19:19:37.631924 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:19:37.631900    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 19:19:37.639742 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:19:37.639713    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 19:19:53.618129 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:19:53.618077    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 19:19:53.618129 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:19:53.618077    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 19:19:54.126358 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:19:54.126323    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 19:19:54.126577 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:19:54.126323    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 19:19:58.308492 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:19:58.308457    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:58.634671 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:19:58.634580    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:58.704266 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:19:58.704199    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:16.975899 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:20:16.975861    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:58.531282 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:20:58.531198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:58.822614 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:20:58.822518    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:00.889514 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:21:00.889467    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:21:01.395353 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:21:01.395318    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:21:03.316203 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:21:03.316167    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:03.397693 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:21:03.397651    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:46.827219 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:21:46.827176    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:53.749696 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:21:53.749662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:54.024076 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:21:54.023965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:57.127905 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:21:57.127860    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 19:21:57.127905 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:21:57.127874    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 19:21:57.627558 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:21:57.627520    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 19:21:57.627786 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:21:57.627522    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 19:22:38.979230 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:22:38.979111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:04.396989 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:23:04.396929    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:23:04.909600 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:23:04.909572    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:23:24.613541 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:23:24.613508    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:24.732191 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:23:24.732154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:24.826997 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:23:24.826965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:30.435210 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:23:30.435169    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:00.628261 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:24:00.628153    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 19:24:00.628736 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:24:00.628395    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 19:24:01.136156 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:24:01.136131    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 19:24:01.136340 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:24:01.136131    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 19:24:08.912367 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:24:08.912325    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:08.977603 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:24:08.977555    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:37.645863 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:24:37.645831    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 19:24:37.654172 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:24:37.654144    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 19:24:37.657330 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:24:37.657307    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 19:24:37.665209 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:24:37.665185    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 19:24:39.362749 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:24:39.362717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:40.272216 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:24:40.272173    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:40.336280 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:24:40.336240    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:07.910612 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:25:07.910561    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:25:08.417137 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:25:08.417102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:25:25.529177 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:25:25.529137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:25.782419 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:25:25.782329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:35.063058 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:25:35.062973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:36.496261 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:25:36.496221    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:36.594094 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:25:36.594057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:52.811262 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:25:52.811222    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:04.137400 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:26:04.137352    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 19:26:04.137400 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:26:04.137352    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 19:26:04.648914 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:26:04.648889    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 19:26:04.649111 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:26:04.648889    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 19:27:11.418765 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:27:11.418658    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:27:11.921898 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:27:11.921860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:28:01.549488 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:01.549456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:28:02.696743 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:02.696706    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:28:07.649703 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:28:07.649652    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" podUID="e2fb5119-9261-41ca-8654-b7060a960639"
Jul 09 19:28:07.649703 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:28:07.649668    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" podUID="f790da47-896e-405c-ba3f-608321ec51e5"
Jul 09 19:28:08.148502 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:08.148465    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft"
Jul 09 19:28:08.148502 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:08.148494    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94"
Jul 09 19:28:21.044548 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:21.044510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:28:21.136747 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:21.136710    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:28:37.144805 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:37.144758    2578 ???:1] "http: TLS handshake error from 10.0.137.79:45866: EOF"
Jul 09 19:28:37.270536 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:37.270507    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:28:37.270755 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:37.270728    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:28:38.914287 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:38.914254    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:28:38.914688 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:38.914606    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:28:40.524861 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:40.524826    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:28:40.525303 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:40.525124    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:28:42.139623 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:42.139594    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:28:42.140127 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:42.139914    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:28:43.803330 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:43.803300    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:28:43.803765 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:43.803511    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:28:45.421883 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:45.421853    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:28:45.422278 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:45.422176    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:28:47.021283 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:47.021252    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:28:47.025490 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:47.025470    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:28:48.735688 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:48.735652    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:28:48.736198 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:48.735843    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:28:50.513836 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:50.513779    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:28:50.514287 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:50.514105    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:28:52.151963 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:52.151923    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:28:52.152352 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:52.152218    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:28:53.786172 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:53.786131    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:28:53.786627 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:53.786426    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:28:55.387510 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:55.387475    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:28:55.388018 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:55.387726    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:28:56.982814 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:56.982767    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:28:56.983259 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:56.982969    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:28:58.578375 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:58.578336    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:28:58.578906 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:28:58.578621    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:29:00.211773 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:00.211728    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:29:00.212248 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:00.211934    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:29:01.818324 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:01.818288    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/1.log"
Jul 09 19:29:01.819010 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:01.818631    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-r8qw5_44a89107-00ea-4275-ab07-7291b5b7e3c4/istio-proxy/0.log"
Jul 09 19:29:03.680186 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:03.680142    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-675d564c74-s5fsx_c5c938ef-4491-4fc7-aacb-02fa8e3ce61b/router/0.log"
Jul 09 19:29:05.038985 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:05.038944    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-675d564c74-s5fsx_c5c938ef-4491-4fc7-aacb-02fa8e3ce61b/router/0.log"
Jul 09 19:29:06.364765 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:06.364728    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-675d564c74-s5fsx_c5c938ef-4491-4fc7-aacb-02fa8e3ce61b/router/0.log"
Jul 09 19:29:07.674588 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:07.674558    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-675d564c74-s5fsx_c5c938ef-4491-4fc7-aacb-02fa8e3ce61b/router/0.log"
Jul 09 19:29:08.638484 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:08.638432    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:08.683341 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:08.683313    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:09.872523 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:09.872478    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:09.914846 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:09.914789    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:11.089897 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:11.089858    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:11.133039 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:11.133014    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:12.298864 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:12.298826    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:12.346527 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:12.346505    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:13.537847 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:13.537788    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:13.597142 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:13.597112    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:14.881021 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:14.880985    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:14.923146 ip-10-0-137-79 kubenswrapper[2578]: E0709 19:29:14.923102    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" podUID="a536c660-613b-4de7-9415-ccdf91ae856f"
Jul 09 19:29:14.924567 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:14.924537    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:15.425344 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:15.425311    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4"
Jul 09 19:29:16.141438 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:16.141402    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:16.184207 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:16.184176    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:17.354247 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:17.354213    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:17.398474 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:17.398439    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:18.574284 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:18.574246    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:18.624117 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:18.624084    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:19.788774 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:19.788730    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:19.828761 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:19.828732    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:21.000837 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:21.000804    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:21.047407 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:21.047352    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:22.238055 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:22.238021    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:22.286778 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:22.286748    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:23.481711 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:23.481663    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:23.526362 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:23.526335    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:24.688781 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:24.688746    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:24.734455 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:24.734424    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:25.917521 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:25.917490    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:25.960040 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:25.960010    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:27.125136 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:27.125103    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:27.168988 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:27.168961    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:28.340156 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:28.340120    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:28.384936 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:28.384906    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:29.547170 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:29.547115    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:29.594688 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:29.594662    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:30.766766 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:30.766719    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:30.812627 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:30.812599    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:31.982843 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:31.982811    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:32.024301 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:32.024273    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:33.219097 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:33.219063    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:33.267060 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:33.267030    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:36.873846 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:36.873791    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:29:36.977591 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:36.977552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:29:37.671077 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:37.671049    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 19:29:37.679672 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:37.679638    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 19:29:37.683656 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:37.683633    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 19:29:37.691227 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:37.691203    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlk4c_555b59d8-7ef9-4d32-bfe4-b3ec61808eba/ovn-acl-logging/0.log"
Jul 09 19:29:39.034838 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:39.034780    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5kvp7_d3da572d-acbd-4c3c-a693-c0eac330f719/global-pull-secret-syncer/0.log"
Jul 09 19:29:39.138178 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:39.138139    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log"
Jul 09 19:29:39.316526 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:39.316419    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-79.ec2.internal_fb2f2f9ee8258d42945cd1eb17f2050c/haproxy/0.log"
Jul 09 19:29:42.322445 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:42.322393    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:29:42.424746 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:42.424711    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:29:43.667163 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:43.667070    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-87vr8_2c3b6827-03a1-4b78-867f-a31dd2fbbf64/authorino/0.log"
Jul 09 19:29:43.768694 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:43.768666    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7mvzh_0a93127a-15c3-410c-9f2b-1102ee005f45/kuadrant-console-plugin/0.log"
Jul 09 19:29:45.636737 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:45.636703    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-cjt5m_1d25c758-d9bb-4113-a71d-de846c62e5ef/cluster-monitoring-operator/0.log"
Jul 09 19:29:45.796982 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:45.796953    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g7js5_3b635a0d-0095-471e-8ff4-d47c7a234223/node-exporter/0.log"
Jul 09 19:29:45.821861 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:45.821834    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g7js5_3b635a0d-0095-471e-8ff4-d47c7a234223/kube-rbac-proxy/0.log"
Jul 09 19:29:45.843809 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:45.843773    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g7js5_3b635a0d-0095-471e-8ff4-d47c7a234223/init-textfile/0.log"
Jul 09 19:29:46.646519 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:46.646438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42brv_8e441cb9-53b1-41e0-8f86-c0d30c8dd55f/konnectivity-agent/0.log" limit=1024
Jul 09 19:29:48.097383 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:48.097334    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-9zf8g_fccda6c1-c857-41be-a4fd-198cbd82a2c5/networking-console-plugin/0.log"
Jul 09 19:29:48.698945 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:48.698910    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/1.log"
Jul 09 19:29:48.704151 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:48.704131    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kln87_806524a3-c51c-49e3-822e-c04ed12d98cd/console-operator/2.log"
Jul 09 19:29:49.437070 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.437031    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"]
Jul 09 19:29:49.437458 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.437366    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main"
Jul 09 19:29:49.437458 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.437376    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main"
Jul 09 19:29:49.437458 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.437389    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="storage-initializer"
Jul 09 19:29:49.437458 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.437396    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="storage-initializer"
Jul 09 19:29:49.437458 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.437458    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99a8b35d-132c-4f40-834b-8e546707a7f4" containerName="main"
Jul 09 19:29:49.440486 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.440468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.443171 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.443132    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gxtmp\"/\"default-dockercfg-vf5cc\""
Jul 09 19:29:49.443317 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.443202    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gxtmp\"/\"openshift-service-ca.crt\""
Jul 09 19:29:49.444738 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.444704    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gxtmp\"/\"kube-root-ca.crt\""
Jul 09 19:29:49.449934 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.449911    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"]
Jul 09 19:29:49.527178 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.527137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96e8b8b3-7902-4c28-b096-6d4c5cdee010-podres\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.527178 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.527183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96e8b8b3-7902-4c28-b096-6d4c5cdee010-proc\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.527480 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.527255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96e8b8b3-7902-4c28-b096-6d4c5cdee010-sys\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.527480 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.527365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph9cb\" (UniqueName: \"kubernetes.io/projected/96e8b8b3-7902-4c28-b096-6d4c5cdee010-kube-api-access-ph9cb\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.527480 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.527396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96e8b8b3-7902-4c28-b096-6d4c5cdee010-lib-modules\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.628288 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.628239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96e8b8b3-7902-4c28-b096-6d4c5cdee010-podres\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.628288 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.628282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96e8b8b3-7902-4c28-b096-6d4c5cdee010-proc\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.628507 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.628307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96e8b8b3-7902-4c28-b096-6d4c5cdee010-sys\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.628507 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.628360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9cb\" (UniqueName: \"kubernetes.io/projected/96e8b8b3-7902-4c28-b096-6d4c5cdee010-kube-api-access-ph9cb\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.628507 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.628382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96e8b8b3-7902-4c28-b096-6d4c5cdee010-lib-modules\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.628507 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.628391    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96e8b8b3-7902-4c28-b096-6d4c5cdee010-proc\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.628507 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.628429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96e8b8b3-7902-4c28-b096-6d4c5cdee010-podres\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.628507 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.628468    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96e8b8b3-7902-4c28-b096-6d4c5cdee010-sys\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.628507 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.628502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96e8b8b3-7902-4c28-b096-6d4c5cdee010-lib-modules\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.636735 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.636704    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph9cb\" (UniqueName: \"kubernetes.io/projected/96e8b8b3-7902-4c28-b096-6d4c5cdee010-kube-api-access-ph9cb\") pod \"perf-node-gather-daemonset-s5htg\" (UID: \"96e8b8b3-7902-4c28-b096-6d4c5cdee010\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.752439 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.752403    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:49.820078 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.820034    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-pf6x5_7bb277a6-db3b-46bb-b1d1-ffca6119ec59/volume-data-source-validator/0.log"
Jul 09 19:29:49.888910 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:49.888883    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"]
Jul 09 19:29:50.570977 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:50.570936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg" event={"ID":"96e8b8b3-7902-4c28-b096-6d4c5cdee010","Type":"ContainerStarted","Data":"b11330745d0ed068caeeb165b14426684e871008c705bf2af37ebba22a14d0ab"}
Jul 09 19:29:50.571437 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:50.570992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg" event={"ID":"96e8b8b3-7902-4c28-b096-6d4c5cdee010","Type":"ContainerStarted","Data":"9e08246b90ecce5d062cb52fc55ccad17a210604170f81cca97e00c4078c89e9"}
Jul 09 19:29:50.571437 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:50.571070    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:50.589299 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:50.589246    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg" podStartSLOduration=1.5892294169999999 podStartE2EDuration="1.589229417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:29:50.587420098 +0000 UTC m=+5113.774397866" watchObservedRunningTime="2026-07-09 19:29:50.589229417 +0000 UTC m=+5113.776207185"
Jul 09 19:29:50.928855 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:50.928729    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p7zkv_122683df-b7fe-4c5a-9974-987713ba1ee2/dns/0.log"
Jul 09 19:29:50.952080 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:50.952054    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p7zkv_122683df-b7fe-4c5a-9974-987713ba1ee2/kube-rbac-proxy/0.log"
Jul 09 19:29:51.127379 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:51.127351    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d7pqh_9ccfb877-5ee9-40fa-a28d-8310e8922463/dns-node-resolver/0.log"
Jul 09 19:29:51.709678 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:51.709647    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-klwxw_c290aebc-bdd2-47e7-a307-f00b1df88b5f/node-ca/0.log"
Jul 09 19:29:53.283752 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:53.283720    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-675d564c74-s5fsx_c5c938ef-4491-4fc7-aacb-02fa8e3ce61b/router/0.log"
Jul 09 19:29:53.841308 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:53.841277    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b56mt_2c211d21-6c7a-4c79-a433-751b85d263b4/serve-healthcheck-canary/0.log"
Jul 09 19:29:54.361037 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:54.361003    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-ccd5k_5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b/insights-operator/0.log"
Jul 09 19:29:54.363360 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:54.363340    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-ccd5k_5a9404a1-b4e7-43e5-b1c5-6eec92b2df4b/insights-operator/1.log"
Jul 09 19:29:54.387034 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:54.387006    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dnhg8_f7b733c6-d68b-4be5-af36-4c7f7d8cddf0/kube-rbac-proxy/0.log"
Jul 09 19:29:54.410596 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:54.410565    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dnhg8_f7b733c6-d68b-4be5-af36-4c7f7d8cddf0/exporter/0.log"
Jul 09 19:29:54.432990 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:54.432961    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dnhg8_f7b733c6-d68b-4be5-af36-4c7f7d8cddf0/extractor/0.log"
Jul 09 19:29:56.585924 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:56.585900    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-s5htg"
Jul 09 19:29:57.452559 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:57.452528    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-9fc7df7df-67cff_d72e52ed-cf6e-486f-bb03-b336eb2272a0/manager/0.log"
Jul 09 19:29:57.505182 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:29:57.505128    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-x6d8k_316c9b2c-218d-4cfe-a844-1a7f45025513/openshift-lws-operator/0.log"
Jul 09 19:30:00.992693 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:30:00.992656    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-675d564c74-s5fsx_c5c938ef-4491-4fc7-aacb-02fa8e3ce61b/router/0.log"
Jul 09 19:30:02.648064 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:30:02.648037    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-675d564c74-s5fsx_c5c938ef-4491-4fc7-aacb-02fa8e3ce61b/router/0.log"
Jul 09 19:30:04.924598 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:30:04.924563    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-54gvz_1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e/kube-storage-version-migrator-operator/1.log"
Jul 09 19:30:04.925752 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:30:04.925728    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-54gvz_1b6be2b5-a1e6-407a-bbf7-3aecabb2b99e/kube-storage-version-migrator-operator/0.log"
Jul 09 19:30:06.121091 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:30:06.121062    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8qxqd_8a408701-9b8b-48a1-b843-165cb5758437/kube-multus/0.log"
Jul 09 19:30:06.327048 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:30:06.327019    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8jmw8_d9f223b1-0765-46bb-b8af-136743836d8a/kube-multus-additional-cni-plugins/0.log"
Jul 09 19:30:06.351571 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:30:06.351543    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8jmw8_d9f223b1-0765-46bb-b8af-136743836d8a/egress-router-binary-copy/0.log"
Jul 09 19:30:06.379478 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:30:06.379442    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8jmw8_d9f223b1-0765-46bb-b8af-136743836d8a/cni-plugins/0.log"
Jul 09 19:30:06.402619 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:30:06.402590    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8jmw8_d9f223b1-0765-46bb-b8af-136743836d8a/bond-cni-plugin/0.log"
Jul 09 19:30:06.426867 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:30:06.426834    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8jmw8_d9f223b1-0765-46bb-b8af-136743836d8a/routeoverride-cni/0.log"
Jul 09 19:30:06.450666 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:30:06.450638    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8jmw8_d9f223b1-0765-46bb-b8af-136743836d8a/whereabouts-cni-bincopy/0.log"
Jul 09 19:30:06.475001 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:30:06.474972    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8jmw8_d9f223b1-0765-46bb-b8af-136743836d8a/whereabouts-cni/0.log"
Jul 09 19:30:06.726500 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:30:06.726474    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kpcpn_11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58/network-metrics-daemon/0.log"
Jul 09 19:30:06.751056 ip-10-0-137-79 kubenswrapper[2578]: I0709 19:30:06.751031    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kpcpn_11a6a28b-c7c4-4cb3-b25b-3d777e5aeb58/kube-rbac-proxy/0.log"