Jul 29 18:32:23.622464 ip-10-0-137-52 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 29 18:32:23.622475 ip-10-0-137-52 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 29 18:32:23.622482 ip-10-0-137-52 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 29 18:32:23.622691 ip-10-0-137-52 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 29 18:32:33.622678 ip-10-0-137-52 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 29 18:32:33.622696 ip-10-0-137-52 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d7696fd0f367409280e7f790ce868b86 --
Jul 29 18:34:32.554646 ip-10-0-137-52 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 18:34:32.923450 ip-10-0-137-52 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 18:34:32.923450 ip-10-0-137-52 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 18:34:32.923450 ip-10-0-137-52 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 18:34:32.923450 ip-10-0-137-52 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 18:34:32.923450 ip-10-0-137-52 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 18:34:32.925629 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.925541    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 18:34:32.930704 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930688    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 18:34:32.930704 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930705    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930709    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930712    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930715    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930718    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930722    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930724    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930731    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930734    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930736    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930739    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930742    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930744    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930747    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930750    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930753    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930755    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930758    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930760    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930763    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 18:34:32.930769 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930766    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930768    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930771    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930774    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930776    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930780    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930782    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930785    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930788    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930791    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930793    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930795    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930798    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930800    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930803    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930806    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930809    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930812    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930814    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 18:34:32.931231 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930818    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930822    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930825    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930827    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930830    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930833    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930835    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930838    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930840    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930843    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930845    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930849    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930853    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930856    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930859    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930863    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930866    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930869    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930874    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 18:34:32.931690 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930878    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930881    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930890    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930894    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930898    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930900    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930903    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930906    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930909    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930911    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930914    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930917    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930921    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930923    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930926    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930928    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930931    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930934    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930936    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930939    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 18:34:32.932148 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930942    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930944    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930946    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930950    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930953    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930956    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930958    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930961    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930964    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930967    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930969    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930972    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930974    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930977    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930980    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930982    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930985    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930987    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930989    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.930992    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 18:34:32.932646 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931685    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931691    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931695    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931697    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931701    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931707    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931710    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931713    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931716    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931718    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931721    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931724    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931727    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931730    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931733    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931735    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931738    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931740    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931743    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931745    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 18:34:32.933116 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931751    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931754    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931757    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931759    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931762    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931765    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931767    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931772    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931775    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931778    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931781    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931784    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931787    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931789    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931792    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931795    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931798    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931801    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931804    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 18:34:32.933620 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931806    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931809    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931812    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931814    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931817    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931820    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931822    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931825    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931827    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931830    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931832    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931836    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931840    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931845    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931849    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931853    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931856    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931860    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931863    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931868    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 18:34:32.934070 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931872    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931876    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931880    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931882    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931885    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931887    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931890    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931893    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931895    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931898    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931900    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931903    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931907    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931909    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931912    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931914    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931917    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931920    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931922    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 18:34:32.934595 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931925    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931928    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931930    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931933    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931935    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931937    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931942    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931945    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931948    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931950    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931953    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931955    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931959    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931961    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931964    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931966    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931969    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931972    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931974    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 18:34:32.935062 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931978    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.931982    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932341    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932346    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932349    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932368    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932371    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932374    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932377    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932381    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932383    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932387    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932389    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932392    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932394    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932397    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932399    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932402    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932405    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 18:34:32.935537 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932409    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932412    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932414    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932417    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932420    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932423    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932425    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932430    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932434    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932437    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932439    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932442    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932445    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932448    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932451    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932454    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932457    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932460    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932463    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932466    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 18:34:32.936019 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932469    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932472    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932474    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932477    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932479    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932482    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932484    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932487    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932489    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932492    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932494    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932497    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932502    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932505    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932508    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932511    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932513    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932516    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932519    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 18:34:32.936524 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932521    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932524    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932526    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932529    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932532    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932534    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932537    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932539    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932542    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932544    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932547    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932549    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932552    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932554    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932557    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932561    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932566    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932570    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932574    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 18:34:32.936986 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932577    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932580    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932583    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932586    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932588    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932591    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932595    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932597    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932600    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932602    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932605    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932608    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932612    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932615    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932617    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932620    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932622    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932625    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932627    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932630    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 18:34:32.937499 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932632    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932635    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932638    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.932640    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932709    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932716    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932723    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932727    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932732    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932736    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932740    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932744    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932747    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932750    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932753    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932756    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932760    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932762    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932766    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932769    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932772    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932775    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932779    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932782    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 18:34:32.937988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932787    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932790    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932793    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932797    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932799    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932803    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932806    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932809    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932812    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932815    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932818    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932821    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932825    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932828    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932830    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932833    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932836    2560 flags.go:64] FLAG: --enable-server="true"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932842    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932847    2560 flags.go:64] FLAG: --event-burst="100"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932850    2560 flags.go:64] FLAG: --event-qps="50"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932853    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932856    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932859    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932862    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 18:34:32.938563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932865    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932868    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932872    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932875    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932878    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932880    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932884    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932886    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932891    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932894    2560 flags.go:64] FLAG: --feature-gates=""
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932897    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932901    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932904    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932907    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932910    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932913    2560 flags.go:64] FLAG: --help="false"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932916    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-137-52.ec2.internal"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932919    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932922    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932925    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932928    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932932    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932935    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 18:34:32.939121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932937    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932940    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932943    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932946    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932949    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932952    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932956    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932958    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932962    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932964    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932967    2560 flags.go:64] FLAG: --lock-file=""
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932971    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932974    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932977    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932982    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932985    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932988    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932992    2560 flags.go:64] FLAG: --logging-format="text"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932995    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.932998    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933001    2560 flags.go:64] FLAG: --manifest-url=""
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933004    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933008    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933011    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933015    2560 flags.go:64] FLAG: --max-pods="110"
Jul 29 18:34:32.939713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933018    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933021    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933024    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933027    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933030    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933034    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933037    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933043    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933046    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933049    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933052    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933056    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933061    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933064    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933067    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933070    2560 flags.go:64] FLAG: --port="10250"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933073    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933076    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b13921654d8f5920"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933079    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933082    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933085    2560 flags.go:64] FLAG: --register-node="true"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933088    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933091    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933094    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 29 18:34:32.940604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933098    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933101    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933105    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933107    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933110    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933114    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933116    2560 flags.go:64] FLAG: --runonce="false"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933120    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933123    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933126    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933129    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933131    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933134    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933137    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933140    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933143    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933147    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933150    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933153    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933156    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933159    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933162    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933167    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933170    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933173    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 18:34:32.941691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933177    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933179    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933184    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933187    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933190    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933192    2560 flags.go:64] FLAG: --v="2"
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933196    2560 flags.go:64] FLAG: --version="false"
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933201    2560 flags.go:64] FLAG: --vmodule=""
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933205    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.933208    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933322    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933326    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933329    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933332    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933335    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933338    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933341    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933343    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933346    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933349    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933363    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933366    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 18:34:32.942761 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933368    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933371    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933374    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933376    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933379    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933382    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933385    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933388    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933390    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933393    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933396    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933398    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933403    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933406    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933408    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933411    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933414    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933418    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933421    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933423    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 18:34:32.943890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933426    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933429    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933432    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933435    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933437    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933440    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933442    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933445    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933448    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933450    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933453    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933455    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933458    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933460    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933463    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933465    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933468    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933472    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933476    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933479    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 18:34:32.944662 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933482    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933484    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933487    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933490    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933494    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933496    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933499    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933501    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933504    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933508    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933510    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933513    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933515    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933518    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933520    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933523    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933525    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933528    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933531    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 18:34:32.945313 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933533    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933536    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933538    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933541    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933544    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933546    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933549    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933551    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933554    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933556    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933559    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933561    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933564    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933567    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933569    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933572    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933574    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933579    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933581    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933584    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 18:34:32.946252 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933586    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 18:34:32.946870 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933589    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 18:34:32.946870 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933595    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 18:34:32.946870 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933598    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 18:34:32.946870 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933601    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 18:34:32.946870 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933604    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 18:34:32.946870 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933607    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 18:34:32.946870 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.933611    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 18:34:32.946870 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.934385    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 18:34:32.946870 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.942842    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 18:34:32.946870 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.942859    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 18:34:32.946870 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.942892    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 18:34:32.946870 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.942902    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 18:34:32.946870 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.942997    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943005    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943010    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943015    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943020    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943024    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943028    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943033    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943037    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943041    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943045    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943049    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943054    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943058    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943063    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943067    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943071    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943077    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943084    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 18:34:32.947347 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943089    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943093    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943098    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943102    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943107    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943111    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943116    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943120    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943124    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943129    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943141    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943146    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943150    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943155    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943159    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943164    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943168    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943172    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943177    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943181    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 18:34:32.947853 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943185    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943190    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943194    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943198    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943202    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943206    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943210    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943214    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943219    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943223    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943228    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943232    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943237    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943241    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943246    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943250    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943254    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943258    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943262    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943266    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 18:34:32.948349 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943271    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943275    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943279    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943297    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943302    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943306    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943310    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943315    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943319    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943323    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943327    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943331    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943336    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943340    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943344    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943349    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943371    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943375    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943379    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 18:34:32.948904 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943385    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943389    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943393    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943398    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943401    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943405    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943409    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943413    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943419    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943424    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943428    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943433    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943436    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943440    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943444    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943448    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943452    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943464    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943468    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 18:34:32.949384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943472    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 18:34:32.949900 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943475    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 18:34:32.949900 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.943483    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 18:34:32.949900 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943718    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 18:34:32.949900 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943726    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 18:34:32.949900 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943731    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 18:34:32.949900 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943736    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 18:34:32.949900 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943742    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 18:34:32.949900 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943746    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 18:34:32.949900 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943751    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 18:34:32.949900 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943755    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 18:34:32.949900 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943760    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 18:34:32.949900 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943764    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 18:34:32.949900 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943768    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 18:34:32.949900 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943773    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943777    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943781    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943786    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943790    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943794    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943798    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943803    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943807    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943812    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943816    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943820    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943825    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943829    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943833    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943838    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943842    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943854    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943859    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 18:34:32.950282 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943863    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943867    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943872    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943876    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943880    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943884    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943888    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943892    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943897    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943901    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943904    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943908    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943912    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943916    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943921    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943925    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943929    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943933    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943938    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943942    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 18:34:32.950753 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943946    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943951    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943955    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943959    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943963    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943968    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943972    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943976    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943981    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943985    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.943989    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944003    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944009    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944016    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944020    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944024    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944031    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944037    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944043    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 18:34:32.951577 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944047    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944051    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944056    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944060    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944064    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944068    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944073    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944077    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944081    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944085    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944089    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944094    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944098    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944103    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944107    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944111    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944115    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944120    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944123    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944127    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 18:34:32.952348 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944131    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 18:34:32.953172 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944136    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 18:34:32.953172 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944141    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 18:34:32.953172 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944145    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 18:34:32.953172 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944149    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 18:34:32.953172 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944160    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 18:34:32.953172 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944165    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 18:34:32.953172 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944169    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 18:34:32.953172 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944173    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 18:34:32.953172 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:32.944178    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 18:34:32.953172 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.944186    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 18:34:32.953172 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.944369    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 18:34:32.953172 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.947156    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 18:34:32.953172 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.948550    2560 server.go:1019] "Starting client certificate rotation"
Jul 29 18:34:32.953172 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.948697    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 18:34:32.953721 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.949575    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 18:34:32.954084 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.954064    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 18:34:32.955242 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.955225    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 18:34:32.963276 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.963259    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 29 18:34:32.966596 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.966582    2560 log.go:25] "Validated CRI v1 image API"
Jul 29 18:34:32.967369 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.967341    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 18:34:32.972424 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.972405    2560 fs.go:135] Filesystem UUIDs: map[3a0d6fa5-b6c3-47cc-b2c8-a350973f9f7c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c1eff596-6ba8-41a7-8eeb-4ed40e7f86d5:/dev/nvme0n1p3]
Jul 29 18:34:32.972513 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.972422    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 18:34:32.977073 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.977054    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 18:34:32.977709 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.977609    2560 manager.go:216] Machine: {Timestamp:2026-07-29 18:34:32.977069806 +0000 UTC m=+0.314912917 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096736 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28dc7875bfa4161a0cd075439fd692 SystemUUID:ec28dc78-75bf-a416-1a0c-d075439fd692 BootID:d7696fd0-f367-4092-80e7-f790ce868b86 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ac:5e:2b:0c:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ac:5e:2b:0c:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:db:fb:d6:b1:58 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 18:34:32.977709 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.977708    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 18:34:32.977799 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.977781    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 18:34:32.979551 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.979525    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 18:34:32.979682 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.979554    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-52.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 18:34:32.980836 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.980826    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 18:34:32.980868 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.980839    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 18:34:32.980868 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.980846    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 18:34:32.981487 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.981477    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 18:34:32.982442 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.982432    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 18:34:32.984493 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.984481    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 18:34:32.984614 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.984604    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 18:34:32.984644 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.984634    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 18:34:32.984670 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.984659    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 18:34:32.984700 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.984673    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 18:34:32.984700 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.984681    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 29 18:34:32.984700 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.984698    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 18:34:32.985077 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.985064    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 18:34:32.985123 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.985083    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 18:34:32.987009 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.986985    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 18:34:32.988789 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.988775    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 18:34:32.988840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.988806    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 18:34:32.990540 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.990521    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 18:34:32.990600 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.990547    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 18:34:32.990600 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.990556    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 18:34:32.990600 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.990565    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 18:34:32.990600 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.990577    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 18:34:32.990600 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.990591    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 18:34:32.990736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.990602    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 18:34:32.990736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.990610    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 18:34:32.990736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.990624    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 18:34:32.990736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.990633    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 18:34:32.990736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.990665    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 18:34:32.990736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.990677    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 18:34:32.990736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.990717    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 18:34:32.990736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.990725    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 18:34:32.994188 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.994171    2560 server.go:1268] "Started kubelet"
Jul 29 18:34:32.994306 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.994255    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 18:34:32.994394 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.994328    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 18:34:32.994461 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.994387    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 18:34:32.995067 ip-10-0-137-52 systemd[1]: Started Kubernetes Kubelet.
Jul 29 18:34:32.996391 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.996350    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 18:34:32.996775 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.996756    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-52.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 18:34:32.996869 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:32.996774    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 18:34:32.996869 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:32.996827    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 18:34:32.996955 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:32.996913    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 18:34:32.998813 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:32.998795    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 18:34:33.002635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.002622    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 18:34:33.002692 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.002630    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 18:34:33.002759 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:33.002694    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-52.ec2.internal\" not found"
Jul 29 18:34:33.003226 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.003191    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 18:34:33.003226 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.003210    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 18:34:33.003226 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.003197    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 18:34:33.003433 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.003328    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 18:34:33.003433 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.003339    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 18:34:33.004194 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.004180    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 18:34:33.004194 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.004193    2560 factory.go:55] Registering systemd factory
Jul 29 18:34:33.004319 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.004200    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 29 18:34:33.004428 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.004414    2560 factory.go:153] Registering CRI-O factory
Jul 29 18:34:33.004428 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.004428    2560 factory.go:223] Registration of the crio container factory successfully
Jul 29 18:34:33.004555 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.004447    2560 factory.go:103] Registering Raw factory
Jul 29 18:34:33.004555 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.004461    2560 manager.go:1196] Started watching for new ooms in manager
Jul 29 18:34:33.004836 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.004825    2560 manager.go:349] Starting recovery of all containers
Jul 29 18:34:33.010320 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:33.010294    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 18:34:33.010459 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:33.010435    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 18:34:33.011197 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:33.010187    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-52.ec2.internal.18c6d66ea48876ff  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-52.ec2.internal,UID:ip-10-0-137-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-52.ec2.internal,},FirstTimestamp:2026-07-29 18:34:32.994133759 +0000 UTC m=+0.331976871,LastTimestamp:2026-07-29 18:34:32.994133759 +0000 UTC m=+0.331976871,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-52.ec2.internal,}"
Jul 29 18:34:33.015874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.015853    2560 manager.go:354] Recovery completed
Jul 29 18:34:33.016975 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.016951    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s49zc"
Jul 29 18:34:33.021066 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.021049    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 18:34:33.022996 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.022982    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 18:34:33.023071 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.023014    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 18:34:33.023071 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.023029    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasSufficientPID"
Jul 29 18:34:33.023610 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.023595    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 18:34:33.023610 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.023605    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 18:34:33.023732 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.023609    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 18:34:33.023732 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.023644    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 18:34:33.024300 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.024269    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 18:34:33.024399 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.024321    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 18:34:33.024399 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.024337    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasSufficientPID"
Jul 29 18:34:33.024981 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:33.024920    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-52.ec2.internal.18c6d66ea640dece  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-52.ec2.internal,UID:ip-10-0-137-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-52.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-52.ec2.internal,},FirstTimestamp:2026-07-29 18:34:33.022996174 +0000 UTC m=+0.360839294,LastTimestamp:2026-07-29 18:34:33.022996174 +0000 UTC m=+0.360839294,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-52.ec2.internal,}"
Jul 29 18:34:33.025386 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.025341    2560 policy_none.go:49] "None policy: Start"
Jul 29 18:34:33.025386 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.025374    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 18:34:33.025386 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.025385    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 18:34:33.026424 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.026415    2560 policy_none.go:47] "Start"
Jul 29 18:34:33.027344 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.027323    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s49zc"
Jul 29 18:34:33.067020 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.066870    2560 manager.go:337] "Starting Device Plugin manager"
Jul 29 18:34:33.067111 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:33.067045    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 18:34:33.067111 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.067057    2560 server.go:87] "Starting device plugin registration server"
Jul 29 18:34:33.067307 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.067294    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 18:34:33.067340 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.067309    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 18:34:33.067430 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.067407    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 18:34:33.067544 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.067498    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 18:34:33.067544 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.067507    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 18:34:33.068023 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:33.067999    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 18:34:33.068090 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:33.068063    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-52.ec2.internal\" not found"
Jul 29 18:34:33.167610 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.167585    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 18:34:33.168985 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.168596    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 18:34:33.168985 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.168621    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 18:34:33.168985 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.168634    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasSufficientPID"
Jul 29 18:34:33.168985 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:33.168667    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.624581 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.624519    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.624581 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:34.624544    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-52.ec2.internal\": node \"ip-10-0-137-52.ec2.internal\" not found"
Jul 29 18:34:34.679961 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.679924    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 18:34:34.681232 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.681211    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 18:34:34.681232 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:34.681221    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-52.ec2.internal\" not found"
Jul 29 18:34:34.681369 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.681252    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 18:34:34.681369 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.681275    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 18:34:34.681369 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:34.681310    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 18:34:34.683535 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.683516    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 18:34:34.781814 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.781778    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-52.ec2.internal"]
Jul 29 18:34:34.781814 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:34.781807    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-52.ec2.internal\" not found"
Jul 29 18:34:34.781989 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.781866    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 18:34:34.783435 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.783418    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 18:34:34.783525 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.783445    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 18:34:34.783525 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.783454    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasSufficientPID"
Jul 29 18:34:34.784765 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.784752    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 18:34:34.784885 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.784870    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.784939 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.784904    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 18:34:34.785474 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.785458    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 18:34:34.785550 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.785489    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 18:34:34.785550 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.785464    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 18:34:34.785550 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.785504    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasSufficientPID"
Jul 29 18:34:34.785550 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.785522    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 18:34:34.785550 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.785545    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasSufficientPID"
Jul 29 18:34:34.786751 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.786737    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.786800 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.786764    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 18:34:34.787441 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.787426    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 18:34:34.787501 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.787453    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 18:34:34.787501 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.787466    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeHasSufficientPID"
Jul 29 18:34:34.801731 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:34.801711    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-52.ec2.internal\" not found" node="ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.805359 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:34.805343    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-52.ec2.internal\" not found" node="ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.849637 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.849616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5cda4b95d9841dcfa4719835180ed2ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal\" (UID: \"5cda4b95d9841dcfa4719835180ed2ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.849713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.849643    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cda4b95d9841dcfa4719835180ed2ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal\" (UID: \"5cda4b95d9841dcfa4719835180ed2ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.849713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.849659    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25ae268d780129888ee17e93df9f46c1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-52.ec2.internal\" (UID: \"25ae268d780129888ee17e93df9f46c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.882162 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:34.882109    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-52.ec2.internal\" not found"
Jul 29 18:34:34.950477 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.950451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cda4b95d9841dcfa4719835180ed2ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal\" (UID: \"5cda4b95d9841dcfa4719835180ed2ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.950574 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.950481    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25ae268d780129888ee17e93df9f46c1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-52.ec2.internal\" (UID: \"25ae268d780129888ee17e93df9f46c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.950574 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.950498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5cda4b95d9841dcfa4719835180ed2ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal\" (UID: \"5cda4b95d9841dcfa4719835180ed2ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.950574 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.950538    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cda4b95d9841dcfa4719835180ed2ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal\" (UID: \"5cda4b95d9841dcfa4719835180ed2ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.950685 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.950598    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25ae268d780129888ee17e93df9f46c1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-52.ec2.internal\" (UID: \"25ae268d780129888ee17e93df9f46c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.950685 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:34.950625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5cda4b95d9841dcfa4719835180ed2ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal\" (UID: \"5cda4b95d9841dcfa4719835180ed2ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:34.982574 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:34.982555    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-52.ec2.internal\" not found"
Jul 29 18:34:35.083055 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:35.083020    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-52.ec2.internal\" not found"
Jul 29 18:34:35.104248 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.104225    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:35.109371 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.109345    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:35.183895 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:35.183814    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-52.ec2.internal\" not found"
Jul 29 18:34:35.284401 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:35.284367    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-52.ec2.internal\" not found"
Jul 29 18:34:35.333793 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.333766    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 18:34:35.385114 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:35.385082    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-52.ec2.internal\" not found"
Jul 29 18:34:35.392254 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.392238    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 18:34:35.392369 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.392353    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 18:34:35.392407 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.392389    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 18:34:35.475736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.475658    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 18:29:34 +0000 UTC" deadline="2028-04-22 22:58:15.41125731 +0000 UTC"
Jul 29 18:34:35.475736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.475688    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15196h23m39.935571557s"
Jul 29 18:34:35.485823 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:35.485803    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-52.ec2.internal\" not found"
Jul 29 18:34:35.586464 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:35.586439    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-52.ec2.internal\" not found"
Jul 29 18:34:35.687322 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:35.687297    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-52.ec2.internal\" not found"
Jul 29 18:34:35.747875 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:35.747838    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cda4b95d9841dcfa4719835180ed2ce.slice/crio-c9ee748f42c6ad85b8394633d5f8e73f68f672fd5d844531f690a92282dc61d2 WatchSource:0}: Error finding container c9ee748f42c6ad85b8394633d5f8e73f68f672fd5d844531f690a92282dc61d2: Status 404 returned error can't find the container with id c9ee748f42c6ad85b8394633d5f8e73f68f672fd5d844531f690a92282dc61d2
Jul 29 18:34:35.751821 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.751802    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 18:34:35.755744 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.755727    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 18:34:35.847669 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.847638    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:35.858888 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.858868    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 18:34:35.860636 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.860624    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal"
Jul 29 18:34:35.871274 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.871258    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 18:34:35.925645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:35.925620    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 18:34:36.329810 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.329779    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 18:34:36.430505 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.430469    2560 apiserver.go:52] "Watching apiserver"
Jul 29 18:34:36.436300 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.436267    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 18:34:36.436650 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.436624    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-tffp8","openshift-dns/node-resolver-2qlfn","openshift-image-registry/node-ca-2qm5s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal","openshift-multus/multus-additional-cni-plugins-hfw8j","openshift-multus/multus-r7k7j","openshift-multus/network-metrics-daemon-2jvrj","openshift-network-diagnostics/network-check-target-44mt4","openshift-network-operator/iptables-alerter-x6xsd","openshift-ovn-kubernetes/ovnkube-node-qlsrd","kube-system/global-pull-secret-syncer-rrsh7","kube-system/konnectivity-agent-6d6sh","kube-system/kube-apiserver-proxy-ip-10-0-137-52.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"]
Jul 29 18:34:36.438234 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.438212    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.440113 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.440082    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:36.440294 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:36.440148    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:34:36.440867 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.440784    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 18:34:36.440867 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.440793    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 18:34:36.440867 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.440802    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 18:34:36.441159 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.440944    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 18:34:36.441159 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.441033    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kd5rp\""
Jul 29 18:34:36.441159 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.441089    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 18:34:36.441407 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.441178    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 18:34:36.441497 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.441447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2qm5s"
Jul 29 18:34:36.442843 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.442822    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.444177 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.444158    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.444774 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.444755    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 18:34:36.444863 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.444815    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zpgb4\""
Jul 29 18:34:36.444922 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.444815    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 18:34:36.445184 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.445038    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mnn6z\""
Jul 29 18:34:36.445184 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.445039    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 18:34:36.445184 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.445083    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 18:34:36.445184 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.445107    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 18:34:36.445373 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.445361    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 18:34:36.446868 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.445858    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.446868 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.446442    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 18:34:36.446868 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.446555    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 18:34:36.446868 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.446558    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nlvld\""
Jul 29 18:34:36.446868 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.446579    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 18:34:36.446868 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.446795    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 18:34:36.446868 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.446831    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 18:34:36.447116 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.447069    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 18:34:36.447587 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.447294    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x6xsd"
Jul 29 18:34:36.448134 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.448118    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 18:34:36.450339 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.448890    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:36.450339 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:36.448969    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:34:36.450339 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.449866    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d5zlt\""
Jul 29 18:34:36.450339 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.450018    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qj7qr\""
Jul 29 18:34:36.450339 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.450123    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 18:34:36.450339 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.450127    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 18:34:36.450339 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.450260    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 18:34:36.452096 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.452076    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2qlfn"
Jul 29 18:34:36.454013 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.453993    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.454447 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.454425    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 18:34:36.454525 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.454454    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-47sht\""
Jul 29 18:34:36.454583 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.454549    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 18:34:36.456076 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.456058    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 18:34:36.456394 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.456377    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 18:34:36.456512 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.456493    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:36.456583 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:36.456558    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:34:36.456932 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.456914    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9g2zt\""
Jul 29 18:34:36.458104 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458087    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6d6sh"
Jul 29 18:34:36.458209 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458101    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-sys-fs\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.458209 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-var-lib-openvswitch\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.458209 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458141    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-etc-openvswitch\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.458209 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-run-ovn\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.458461 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51e71484-cd8e-4585-b7a3-c14fb8374d74-iptables-alerter-script\") pod \"iptables-alerter-x6xsd\" (UID: \"51e71484-cd8e-4585-b7a3-c14fb8374d74\") " pod="openshift-network-operator/iptables-alerter-x6xsd"
Jul 29 18:34:36.458461 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458257    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sxpc\" (UniqueName: \"kubernetes.io/projected/51e71484-cd8e-4585-b7a3-c14fb8374d74-kube-api-access-7sxpc\") pod \"iptables-alerter-x6xsd\" (UID: \"51e71484-cd8e-4585-b7a3-c14fb8374d74\") " pod="openshift-network-operator/iptables-alerter-x6xsd"
Jul 29 18:34:36.458461 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458283    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-os-release\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.458461 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458307    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-run-netns\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.458461 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458333    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/244afbce-35b6-4432-a874-d7081b20c82e-dbus\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:36.458461 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-system-cni-dir\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.458461 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458379    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.458461 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjl2w\" (UniqueName: \"kubernetes.io/projected/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-kube-api-access-tjl2w\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.458461 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458427    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-multus-cni-dir\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.458461 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-etc-kubernetes\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.458874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458487    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-systemd-units\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.458874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458511    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-multus-conf-dir\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.458874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458538    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-run-multus-certs\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.458874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458562    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-kubelet\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.458874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-slash\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.458874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458612    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-run-netns\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.458874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458635    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-cni-bin\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.458874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458658    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/990500cb-5912-4252-b507-f499bae20618-host\") pod \"node-ca-2qm5s\" (UID: \"990500cb-5912-4252-b507-f499bae20618\") " pod="openshift-image-registry/node-ca-2qm5s"
Jul 29 18:34:36.458874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458686    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.458874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhnng\" (UniqueName: \"kubernetes.io/projected/b4e01106-0034-44ad-99dd-f438ef1204c2-kube-api-access-bhnng\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.458874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458759    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-run-ovn-kubernetes\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.458874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-node-log\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.458874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458810    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1072bb12-b457-4240-abf9-fbba51caa986-env-overrides\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.458874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458862    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458896    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458923    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-cnibin\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458945    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1072bb12-b457-4240-abf9-fbba51caa986-ovnkube-script-lib\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.458991    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51e71484-cd8e-4585-b7a3-c14fb8374d74-host-slash\") pod \"iptables-alerter-x6xsd\" (UID: \"51e71484-cd8e-4585-b7a3-c14fb8374d74\") " pod="openshift-network-operator/iptables-alerter-x6xsd"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459015    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-var-lib-cni-multus\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459037    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb20b3c0-7818-479d-8fe3-3c5700547014-multus-daemon-config\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-run-systemd\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459102    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sffsg\" (UniqueName: \"kubernetes.io/projected/1072bb12-b457-4240-abf9-fbba51caa986-kube-api-access-sffsg\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz6tx\" (UniqueName: \"kubernetes.io/projected/fb20b3c0-7818-479d-8fe3-3c5700547014-kube-api-access-sz6tx\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459152    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdq9w\" (UniqueName: \"kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w\") pod \"network-check-target-44mt4\" (UID: \"7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e\") " pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459177    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-multus-socket-dir-parent\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459242    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-cnibin\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459268    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-system-cni-dir\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459290    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-var-lib-cni-bin\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.459545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-device-dir\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-run-openvswitch\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459367    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-cni-netd\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2p7v\" (UniqueName: \"kubernetes.io/projected/990500cb-5912-4252-b507-f499bae20618-kube-api-access-p2p7v\") pod \"node-ca-2qm5s\" (UID: \"990500cb-5912-4252-b507-f499bae20618\") " pod="openshift-image-registry/node-ca-2qm5s"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459467    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-run-k8s-cni-cncf-io\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459595    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-socket-dir\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459643    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-registration-dir\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459700    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-os-release\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459699    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459763    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459792    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb20b3c0-7818-479d-8fe3-3c5700547014-cni-binary-copy\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459808    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-log-socket\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459831    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-var-lib-kubelet\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-hostroot\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459901    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.460148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459941    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1072bb12-b457-4240-abf9-fbba51caa986-ovnkube-config\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.460731 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.459972    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1072bb12-b457-4240-abf9-fbba51caa986-ovn-node-metrics-cert\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.460731 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.460017    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/244afbce-35b6-4432-a874-d7081b20c82e-kubelet-config\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:36.460731 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.460041    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/990500cb-5912-4252-b507-f499bae20618-serviceca\") pod \"node-ca-2qm5s\" (UID: \"990500cb-5912-4252-b507-f499bae20618\") " pod="openshift-image-registry/node-ca-2qm5s"
Jul 29 18:34:36.460731 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.460440    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 18:34:36.460731 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.460598    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 18:34:36.461073 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.461055    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pdt4b\""
Jul 29 18:34:36.482024 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.482001    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9cgqz"
Jul 29 18:34:36.490772 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.490679    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9cgqz"
Jul 29 18:34:36.548083 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.548053    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 18:34:36.560803 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.560772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-var-lib-kubelet\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.560803 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.560801    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-sys-fs\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.560995 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.560821    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-var-lib-openvswitch\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.560995 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.560843    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-etc-openvswitch\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.560995 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.560892    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-run-ovn\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.560995 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.560942    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-run-ovn\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.560995 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.560969    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-sys-fs\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.560995 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.560973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-lib-modules\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.561310 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.560998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-var-lib-openvswitch\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.561310 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561002    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-etc-openvswitch\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.561310 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51e71484-cd8e-4585-b7a3-c14fb8374d74-iptables-alerter-script\") pod \"iptables-alerter-x6xsd\" (UID: \"51e71484-cd8e-4585-b7a3-c14fb8374d74\") " pod="openshift-network-operator/iptables-alerter-x6xsd"
Jul 29 18:34:36.561310 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sxpc\" (UniqueName: \"kubernetes.io/projected/51e71484-cd8e-4585-b7a3-c14fb8374d74-kube-api-access-7sxpc\") pod \"iptables-alerter-x6xsd\" (UID: \"51e71484-cd8e-4585-b7a3-c14fb8374d74\") " pod="openshift-network-operator/iptables-alerter-x6xsd"
Jul 29 18:34:36.561310 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561105    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-os-release\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.561310 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561136    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-run-netns\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.561310 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561215    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/244afbce-35b6-4432-a874-d7081b20c82e-dbus\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:36.561310 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561252    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-system-cni-dir\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.561310 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.561310 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjl2w\" (UniqueName: \"kubernetes.io/projected/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-kube-api-access-tjl2w\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561334    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-multus-cni-dir\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-etc-kubernetes\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-systemd-units\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561410    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/244afbce-35b6-4432-a874-d7081b20c82e-dbus\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561219    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-run-netns\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561412    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-modprobe-d\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561475    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-etc-kubernetes\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-kubernetes\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561219    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-os-release\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561513    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-multus-conf-dir\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-run-multus-certs\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561548    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-systemd-units\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561566    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-kubelet\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561624    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-system-cni-dir\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561627    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-multus-conf-dir\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51e71484-cd8e-4585-b7a3-c14fb8374d74-iptables-alerter-script\") pod \"iptables-alerter-x6xsd\" (UID: \"51e71484-cd8e-4585-b7a3-c14fb8374d74\") " pod="openshift-network-operator/iptables-alerter-x6xsd"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561673    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-run-multus-certs\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.561806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561674    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-kubelet\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561704    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-slash\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561716    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-multus-cni-dir\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561727    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-run-netns\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561748    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561752    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-cni-bin\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561746    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-slash\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561800    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/990500cb-5912-4252-b507-f499bae20618-host\") pod \"node-ca-2qm5s\" (UID: \"990500cb-5912-4252-b507-f499bae20618\") " pod="openshift-image-registry/node-ca-2qm5s"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561807    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-cni-bin\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561805    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-run-netns\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-sys\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561857    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561824    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/990500cb-5912-4252-b507-f499bae20618-host\") pod \"node-ca-2qm5s\" (UID: \"990500cb-5912-4252-b507-f499bae20618\") " pod="openshift-image-registry/node-ca-2qm5s"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561904    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhnng\" (UniqueName: \"kubernetes.io/projected/b4e01106-0034-44ad-99dd-f438ef1204c2-kube-api-access-bhnng\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-run-ovn-kubernetes\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs\") pod \"network-metrics-daemon-2jvrj\" (UID: \"7154111a-1f26-4f41-8ba6-94617dcca1b3\") " pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561975    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.562753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.561998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-run-ovn-kubernetes\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xhvk\" (UniqueName: \"kubernetes.io/projected/7154111a-1f26-4f41-8ba6-94617dcca1b3-kube-api-access-6xhvk\") pod \"network-metrics-daemon-2jvrj\" (UID: \"7154111a-1f26-4f41-8ba6-94617dcca1b3\") " pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c4effb3-c122-47d9-88ad-557670f8248d-agent-certs\") pod \"konnectivity-agent-6d6sh\" (UID: \"2c4effb3-c122-47d9-88ad-557670f8248d\") " pod="kube-system/konnectivity-agent-6d6sh"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562121    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c4effb3-c122-47d9-88ad-557670f8248d-konnectivity-ca\") pod \"konnectivity-agent-6d6sh\" (UID: \"2c4effb3-c122-47d9-88ad-557670f8248d\") " pod="kube-system/konnectivity-agent-6d6sh"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-node-log\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1072bb12-b457-4240-abf9-fbba51caa986-env-overrides\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562208    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-run\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562233    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562237    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-node-log\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562308    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-cnibin\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1072bb12-b457-4240-abf9-fbba51caa986-ovnkube-script-lib\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562374    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562380    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-sysctl-d\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4ea3227-0889-453b-b4af-696d070a0174-etc-tuned\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:36.562465    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:36.563645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562525    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51e71484-cd8e-4585-b7a3-c14fb8374d74-host-slash\") pod \"iptables-alerter-x6xsd\" (UID: \"51e71484-cd8e-4585-b7a3-c14fb8374d74\") " pod="openshift-network-operator/iptables-alerter-x6xsd"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:36.562546    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret podName:244afbce-35b6-4432-a874-d7081b20c82e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:37.062525261 +0000 UTC m=+2.956525639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret") pod "global-pull-secret-syncer-rrsh7" (UID: "244afbce-35b6-4432-a874-d7081b20c82e") : object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51e71484-cd8e-4585-b7a3-c14fb8374d74-host-slash\") pod \"iptables-alerter-x6xsd\" (UID: \"51e71484-cd8e-4585-b7a3-c14fb8374d74\") " pod="openshift-network-operator/iptables-alerter-x6xsd"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-var-lib-cni-multus\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-systemd\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562700    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb20b3c0-7818-479d-8fe3-3c5700547014-multus-daemon-config\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562717    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-run-systemd\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562720    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1072bb12-b457-4240-abf9-fbba51caa986-env-overrides\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sffsg\" (UniqueName: \"kubernetes.io/projected/1072bb12-b457-4240-abf9-fbba51caa986-kube-api-access-sffsg\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c98ed70e-1aab-43f0-b57d-192dac28efd6-hosts-file\") pod \"node-resolver-2qlfn\" (UID: \"c98ed70e-1aab-43f0-b57d-192dac28efd6\") " pod="openshift-dns/node-resolver-2qlfn"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562800    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-sysconfig\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562827    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxrt5\" (UniqueName: \"kubernetes.io/projected/a4ea3227-0889-453b-b4af-696d070a0174-kube-api-access-bxrt5\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562853    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz6tx\" (UniqueName: \"kubernetes.io/projected/fb20b3c0-7818-479d-8fe3-3c5700547014-kube-api-access-sz6tx\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562880    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdq9w\" (UniqueName: \"kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w\") pod \"network-check-target-44mt4\" (UID: \"7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e\") " pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562919    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-multus-socket-dir-parent\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562970    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-var-lib-cni-multus\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.564521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.562967    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-cnibin\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563006    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-system-cni-dir\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563032    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-var-lib-cni-bin\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563067    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-device-dir\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563108    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-run-openvswitch\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563134    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-cni-netd\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563169    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c98ed70e-1aab-43f0-b57d-192dac28efd6-tmp-dir\") pod \"node-resolver-2qlfn\" (UID: \"c98ed70e-1aab-43f0-b57d-192dac28efd6\") " pod="openshift-dns/node-resolver-2qlfn"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563281    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2p7v\" (UniqueName: \"kubernetes.io/projected/990500cb-5912-4252-b507-f499bae20618-kube-api-access-p2p7v\") pod \"node-ca-2qm5s\" (UID: \"990500cb-5912-4252-b507-f499bae20618\") " pod="openshift-image-registry/node-ca-2qm5s"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563305    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4ea3227-0889-453b-b4af-696d070a0174-tmp\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-run-k8s-cni-cncf-io\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563382    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-cnibin\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563387    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-socket-dir\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563436    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-run-systemd\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563443    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-registration-dir\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563495    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-var-lib-cni-bin\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563497    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-socket-dir\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.565334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563542    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-run-openvswitch\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563552    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-os-release\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563570    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-cni-netd\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563311    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1072bb12-b457-4240-abf9-fbba51caa986-ovnkube-script-lib\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563592    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-device-dir\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563456    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb20b3c0-7818-479d-8fe3-3c5700547014-multus-daemon-config\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563670    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-multus-socket-dir-parent\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563671    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-run-k8s-cni-cncf-io\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563704    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-cnibin\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563723    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-system-cni-dir\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563769    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b4e01106-0034-44ad-99dd-f438ef1204c2-registration-dir\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563863    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-os-release\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.563955    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564007    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb20b3c0-7818-479d-8fe3-3c5700547014-cni-binary-copy\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564034    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-log-socket\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564061    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-var-lib-kubelet\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.566070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564083    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-hostroot\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564106    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564112    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-log-socket\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564128    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1072bb12-b457-4240-abf9-fbba51caa986-ovnkube-config\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564151    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1072bb12-b457-4240-abf9-fbba51caa986-ovn-node-metrics-cert\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564177    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj5x4\" (UniqueName: \"kubernetes.io/projected/c98ed70e-1aab-43f0-b57d-192dac28efd6-kube-api-access-hj5x4\") pod \"node-resolver-2qlfn\" (UID: \"c98ed70e-1aab-43f0-b57d-192dac28efd6\") " pod="openshift-dns/node-resolver-2qlfn"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-sysctl-conf\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564287    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/244afbce-35b6-4432-a874-d7081b20c82e-kubelet-config\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564289    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564316    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/990500cb-5912-4252-b507-f499bae20618-serviceca\") pod \"node-ca-2qm5s\" (UID: \"990500cb-5912-4252-b507-f499bae20618\") " pod="openshift-image-registry/node-ca-2qm5s"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564341    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-host\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564464    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-host-var-lib-kubelet\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564544    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb20b3c0-7818-479d-8fe3-3c5700547014-hostroot\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564549    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb20b3c0-7818-479d-8fe3-3c5700547014-cni-binary-copy\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564603    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1072bb12-b457-4240-abf9-fbba51caa986-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.564597    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.565013    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 18:34:36.566840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.565665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/244afbce-35b6-4432-a874-d7081b20c82e-kubelet-config\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:36.567572 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.565731    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1072bb12-b457-4240-abf9-fbba51caa986-ovnkube-config\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.567572 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.566116    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/990500cb-5912-4252-b507-f499bae20618-serviceca\") pod \"node-ca-2qm5s\" (UID: \"990500cb-5912-4252-b507-f499bae20618\") " pod="openshift-image-registry/node-ca-2qm5s"
Jul 29 18:34:36.569333 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.569275    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1072bb12-b457-4240-abf9-fbba51caa986-ovn-node-metrics-cert\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.570787 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:36.570766    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 18:34:36.570787 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:36.570789    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 18:34:36.571035 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:36.570803    2560 projected.go:196] Error preparing data for projected volume kube-api-access-bdq9w for pod openshift-network-diagnostics/network-check-target-44mt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:36.571035 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:36.570855    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w podName:7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:37.070838676 +0000 UTC m=+2.964839059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bdq9w" (UniqueName: "kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w") pod "network-check-target-44mt4" (UID: "7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:36.571547 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.571510    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sxpc\" (UniqueName: \"kubernetes.io/projected/51e71484-cd8e-4585-b7a3-c14fb8374d74-kube-api-access-7sxpc\") pod \"iptables-alerter-x6xsd\" (UID: \"51e71484-cd8e-4585-b7a3-c14fb8374d74\") " pod="openshift-network-operator/iptables-alerter-x6xsd"
Jul 29 18:34:36.571655 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.571624    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjl2w\" (UniqueName: \"kubernetes.io/projected/396aaa29-be70-4f0b-bd94-a3cc455a5cd2-kube-api-access-tjl2w\") pod \"multus-additional-cni-plugins-hfw8j\" (UID: \"396aaa29-be70-4f0b-bd94-a3cc455a5cd2\") " pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.572514 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.572492    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sffsg\" (UniqueName: \"kubernetes.io/projected/1072bb12-b457-4240-abf9-fbba51caa986-kube-api-access-sffsg\") pod \"ovnkube-node-qlsrd\" (UID: \"1072bb12-b457-4240-abf9-fbba51caa986\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.572960 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.572941    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhnng\" (UniqueName: \"kubernetes.io/projected/b4e01106-0034-44ad-99dd-f438ef1204c2-kube-api-access-bhnng\") pod \"aws-ebs-csi-driver-node-cjp66\" (UID: \"b4e01106-0034-44ad-99dd-f438ef1204c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.573713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.573692    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz6tx\" (UniqueName: \"kubernetes.io/projected/fb20b3c0-7818-479d-8fe3-3c5700547014-kube-api-access-sz6tx\") pod \"multus-r7k7j\" (UID: \"fb20b3c0-7818-479d-8fe3-3c5700547014\") " pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.573713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.573692    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2p7v\" (UniqueName: \"kubernetes.io/projected/990500cb-5912-4252-b507-f499bae20618-kube-api-access-p2p7v\") pod \"node-ca-2qm5s\" (UID: \"990500cb-5912-4252-b507-f499bae20618\") " pod="openshift-image-registry/node-ca-2qm5s"
Jul 29 18:34:36.665625 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665551    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-host\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.665625 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665587    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-var-lib-kubelet\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.665625 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665610    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-lib-modules\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.665879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-host\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.665879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665725    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-modprobe-d\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.665879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665749    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-kubernetes\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.665879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-lib-modules\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.665879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665774    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-sys\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.665879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665812    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs\") pod \"network-metrics-daemon-2jvrj\" (UID: \"7154111a-1f26-4f41-8ba6-94617dcca1b3\") " pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:36.665879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665818    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-modprobe-d\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.665879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665823    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-kubernetes\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.665879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665832    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhvk\" (UniqueName: \"kubernetes.io/projected/7154111a-1f26-4f41-8ba6-94617dcca1b3-kube-api-access-6xhvk\") pod \"network-metrics-daemon-2jvrj\" (UID: \"7154111a-1f26-4f41-8ba6-94617dcca1b3\") " pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:36.665879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665858    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c4effb3-c122-47d9-88ad-557670f8248d-agent-certs\") pod \"konnectivity-agent-6d6sh\" (UID: \"2c4effb3-c122-47d9-88ad-557670f8248d\") " pod="kube-system/konnectivity-agent-6d6sh"
Jul 29 18:34:36.665879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665705    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-var-lib-kubelet\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.665879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665875    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-sys\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.665879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665882    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c4effb3-c122-47d9-88ad-557670f8248d-konnectivity-ca\") pod \"konnectivity-agent-6d6sh\" (UID: \"2c4effb3-c122-47d9-88ad-557670f8248d\") " pod="kube-system/konnectivity-agent-6d6sh"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665908    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-run\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665946    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-sysctl-d\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.665966    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4ea3227-0889-453b-b4af-696d070a0174-etc-tuned\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:36.666014    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666024    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-systemd\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666035    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-run\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c98ed70e-1aab-43f0-b57d-192dac28efd6-hosts-file\") pod \"node-resolver-2qlfn\" (UID: \"c98ed70e-1aab-43f0-b57d-192dac28efd6\") " pod="openshift-dns/node-resolver-2qlfn"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:36.666083    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs podName:7154111a-1f26-4f41-8ba6-94617dcca1b3 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:37.166062113 +0000 UTC m=+3.060062503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs") pod "network-metrics-daemon-2jvrj" (UID: "7154111a-1f26-4f41-8ba6-94617dcca1b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666102    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-sysconfig\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666111    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c98ed70e-1aab-43f0-b57d-192dac28efd6-hosts-file\") pod \"node-resolver-2qlfn\" (UID: \"c98ed70e-1aab-43f0-b57d-192dac28efd6\") " pod="openshift-dns/node-resolver-2qlfn"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666130    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxrt5\" (UniqueName: \"kubernetes.io/projected/a4ea3227-0889-453b-b4af-696d070a0174-kube-api-access-bxrt5\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666133    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-systemd\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666174    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c98ed70e-1aab-43f0-b57d-192dac28efd6-tmp-dir\") pod \"node-resolver-2qlfn\" (UID: \"c98ed70e-1aab-43f0-b57d-192dac28efd6\") " pod="openshift-dns/node-resolver-2qlfn"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666215    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4ea3227-0889-453b-b4af-696d070a0174-tmp\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666241    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-sysctl-d\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj5x4\" (UniqueName: \"kubernetes.io/projected/c98ed70e-1aab-43f0-b57d-192dac28efd6-kube-api-access-hj5x4\") pod \"node-resolver-2qlfn\" (UID: \"c98ed70e-1aab-43f0-b57d-192dac28efd6\") " pod="openshift-dns/node-resolver-2qlfn"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666277    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-sysctl-conf\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.666476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-sysconfig\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.667216 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4ea3227-0889-453b-b4af-696d070a0174-etc-sysctl-conf\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.667216 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666620    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c4effb3-c122-47d9-88ad-557670f8248d-konnectivity-ca\") pod \"konnectivity-agent-6d6sh\" (UID: \"2c4effb3-c122-47d9-88ad-557670f8248d\") " pod="kube-system/konnectivity-agent-6d6sh"
Jul 29 18:34:36.667216 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.666645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c98ed70e-1aab-43f0-b57d-192dac28efd6-tmp-dir\") pod \"node-resolver-2qlfn\" (UID: \"c98ed70e-1aab-43f0-b57d-192dac28efd6\") " pod="openshift-dns/node-resolver-2qlfn"
Jul 29 18:34:36.668761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.668737    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4ea3227-0889-453b-b4af-696d070a0174-etc-tuned\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.668878 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.668795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c4effb3-c122-47d9-88ad-557670f8248d-agent-certs\") pod \"konnectivity-agent-6d6sh\" (UID: \"2c4effb3-c122-47d9-88ad-557670f8248d\") " pod="kube-system/konnectivity-agent-6d6sh"
Jul 29 18:34:36.668940 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.668918    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4ea3227-0889-453b-b4af-696d070a0174-tmp\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.677502 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.677471    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxrt5\" (UniqueName: \"kubernetes.io/projected/a4ea3227-0889-453b-b4af-696d070a0174-kube-api-access-bxrt5\") pod \"tuned-tffp8\" (UID: \"a4ea3227-0889-453b-b4af-696d070a0174\") " pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.677853 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.677828    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj5x4\" (UniqueName: \"kubernetes.io/projected/c98ed70e-1aab-43f0-b57d-192dac28efd6-kube-api-access-hj5x4\") pod \"node-resolver-2qlfn\" (UID: \"c98ed70e-1aab-43f0-b57d-192dac28efd6\") " pod="openshift-dns/node-resolver-2qlfn"
Jul 29 18:34:36.678240 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.678188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xhvk\" (UniqueName: \"kubernetes.io/projected/7154111a-1f26-4f41-8ba6-94617dcca1b3-kube-api-access-6xhvk\") pod \"network-metrics-daemon-2jvrj\" (UID: \"7154111a-1f26-4f41-8ba6-94617dcca1b3\") " pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:36.686096 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.686050    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal" event={"ID":"5cda4b95d9841dcfa4719835180ed2ce","Type":"ContainerStarted","Data":"c9ee748f42c6ad85b8394633d5f8e73f68f672fd5d844531f690a92282dc61d2"}
Jul 29 18:34:36.750486 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.750450    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:34:36.757921 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.757898    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2qm5s"
Jul 29 18:34:36.766789 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.766762    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66"
Jul 29 18:34:36.773588 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.773566    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6d6sh"
Jul 29 18:34:36.779376 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.779348    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hfw8j"
Jul 29 18:34:36.784987 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.784967    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r7k7j"
Jul 29 18:34:36.792739 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.792719    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x6xsd"
Jul 29 18:34:36.798433 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.798409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2qlfn"
Jul 29 18:34:36.802099 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:36.802080    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tffp8"
Jul 29 18:34:36.829860 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:36.829832    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98ed70e_1aab_43f0_b57d_192dac28efd6.slice/crio-ce44f7079acbb2a1d112ccfab04028931b66f770e8221f040b524bf02707eb91 WatchSource:0}: Error finding container ce44f7079acbb2a1d112ccfab04028931b66f770e8221f040b524bf02707eb91: Status 404 returned error can't find the container with id ce44f7079acbb2a1d112ccfab04028931b66f770e8221f040b524bf02707eb91
Jul 29 18:34:36.830916 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:36.830884    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990500cb_5912_4252_b507_f499bae20618.slice/crio-e8cd4a85a40655af63058862e8dbcbe81050e46616c95a14aa95919fa7851ea5 WatchSource:0}: Error finding container e8cd4a85a40655af63058862e8dbcbe81050e46616c95a14aa95919fa7851ea5: Status 404 returned error can't find the container with id e8cd4a85a40655af63058862e8dbcbe81050e46616c95a14aa95919fa7851ea5
Jul 29 18:34:36.832003 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:36.831976    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1072bb12_b457_4240_abf9_fbba51caa986.slice/crio-74514c0cb314803739e59f0e3b8e95c59088790de7102e74c202452de1fd9e23 WatchSource:0}: Error finding container 74514c0cb314803739e59f0e3b8e95c59088790de7102e74c202452de1fd9e23: Status 404 returned error can't find the container with id 74514c0cb314803739e59f0e3b8e95c59088790de7102e74c202452de1fd9e23
Jul 29 18:34:36.852380 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:36.852354    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e01106_0034_44ad_99dd_f438ef1204c2.slice/crio-dc0b08eca2ba7c4a14968d266b37b7563ebfc74b76d867efb65547dffb138699 WatchSource:0}: Error finding container dc0b08eca2ba7c4a14968d266b37b7563ebfc74b76d867efb65547dffb138699: Status 404 returned error can't find the container with id dc0b08eca2ba7c4a14968d266b37b7563ebfc74b76d867efb65547dffb138699
Jul 29 18:34:36.854058 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:36.854037    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4effb3_c122_47d9_88ad_557670f8248d.slice/crio-45630cd74b3287e38791f045b85493161836faf7885be1f92d7a8a8ebacd257a WatchSource:0}: Error finding container 45630cd74b3287e38791f045b85493161836faf7885be1f92d7a8a8ebacd257a: Status 404 returned error can't find the container with id 45630cd74b3287e38791f045b85493161836faf7885be1f92d7a8a8ebacd257a
Jul 29 18:34:36.854928 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:36.854904    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e71484_cd8e_4585_b7a3_c14fb8374d74.slice/crio-09c0ce1a92bfca4b0072899a0f60fa3370a6b46775c6bf96109ea3214e6f64a0 WatchSource:0}: Error finding container 09c0ce1a92bfca4b0072899a0f60fa3370a6b46775c6bf96109ea3214e6f64a0: Status 404 returned error can't find the container with id 09c0ce1a92bfca4b0072899a0f60fa3370a6b46775c6bf96109ea3214e6f64a0
Jul 29 18:34:36.855913 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:36.855870    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ea3227_0889_453b_b4af_696d070a0174.slice/crio-7db6d112cccfe41473556cf722d38eec5c85f3af51044a73ad27d71f722cceb8 WatchSource:0}: Error finding container 7db6d112cccfe41473556cf722d38eec5c85f3af51044a73ad27d71f722cceb8: Status 404 returned error can't find the container with id 7db6d112cccfe41473556cf722d38eec5c85f3af51044a73ad27d71f722cceb8
Jul 29 18:34:36.856763 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:36.856741    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396aaa29_be70_4f0b_bd94_a3cc455a5cd2.slice/crio-4a81373a0aa0467cbf556bdf39fb4504425d73cc80431c9941fbb7f48b0d5582 WatchSource:0}: Error finding container 4a81373a0aa0467cbf556bdf39fb4504425d73cc80431c9941fbb7f48b0d5582: Status 404 returned error can't find the container with id 4a81373a0aa0467cbf556bdf39fb4504425d73cc80431c9941fbb7f48b0d5582
Jul 29 18:34:36.857316 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:36.857300    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb20b3c0_7818_479d_8fe3_3c5700547014.slice/crio-ace7d04f320b01cdf7c0c9611f46a899239e9317c6dd211dd915dc88dad877fe WatchSource:0}: Error finding container ace7d04f320b01cdf7c0c9611f46a899239e9317c6dd211dd915dc88dad877fe: Status 404 returned error can't find the container with id ace7d04f320b01cdf7c0c9611f46a899239e9317c6dd211dd915dc88dad877fe
Jul 29 18:34:36.877890 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:34:36.877863    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ae268d780129888ee17e93df9f46c1.slice/crio-9c89533d570ec21c2f6a48baa2cfacbad1100b43f1768f154381a9b382b4f558 WatchSource:0}: Error finding container 9c89533d570ec21c2f6a48baa2cfacbad1100b43f1768f154381a9b382b4f558: Status 404 returned error can't find the container with id 9c89533d570ec21c2f6a48baa2cfacbad1100b43f1768f154381a9b382b4f558
Jul 29 18:34:37.069082 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.069051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:37.069241 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:37.069222    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:37.069305 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:37.069295    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret podName:244afbce-35b6-4432-a874-d7081b20c82e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:38.069273789 +0000 UTC m=+3.963274183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret") pod "global-pull-secret-syncer-rrsh7" (UID: "244afbce-35b6-4432-a874-d7081b20c82e") : object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:37.169732 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.169700    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs\") pod \"network-metrics-daemon-2jvrj\" (UID: \"7154111a-1f26-4f41-8ba6-94617dcca1b3\") " pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:37.169889 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.169762    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdq9w\" (UniqueName: \"kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w\") pod \"network-check-target-44mt4\" (UID: \"7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e\") " pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:37.169889 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:37.169842    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:37.170050 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:37.169900    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 18:34:37.170050 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:37.169919    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 18:34:37.170050 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:37.169932    2560 projected.go:196] Error preparing data for projected volume kube-api-access-bdq9w for pod openshift-network-diagnostics/network-check-target-44mt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:37.170050 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:37.169906    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs podName:7154111a-1f26-4f41-8ba6-94617dcca1b3 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:38.169890185 +0000 UTC m=+4.063890565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs") pod "network-metrics-daemon-2jvrj" (UID: "7154111a-1f26-4f41-8ba6-94617dcca1b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:37.170050 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:37.169994    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w podName:7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:38.16997899 +0000 UTC m=+4.063979376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bdq9w" (UniqueName: "kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w") pod "network-check-target-44mt4" (UID: "7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:37.493511 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.493441    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 18:29:36 +0000 UTC" deadline="2028-02-13 17:22:02.591523124 +0000 UTC"
Jul 29 18:34:37.493511 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.493509    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13534h47m25.098018516s"
Jul 29 18:34:37.682410 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.682373    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:37.682583 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:37.682505    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:34:37.699547 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.699505    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-52.ec2.internal" event={"ID":"25ae268d780129888ee17e93df9f46c1","Type":"ContainerStarted","Data":"9c89533d570ec21c2f6a48baa2cfacbad1100b43f1768f154381a9b382b4f558"}
Jul 29 18:34:37.701319 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.701294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r7k7j" event={"ID":"fb20b3c0-7818-479d-8fe3-3c5700547014","Type":"ContainerStarted","Data":"ace7d04f320b01cdf7c0c9611f46a899239e9317c6dd211dd915dc88dad877fe"}
Jul 29 18:34:37.710240 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.710212    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6d6sh" event={"ID":"2c4effb3-c122-47d9-88ad-557670f8248d","Type":"ContainerStarted","Data":"45630cd74b3287e38791f045b85493161836faf7885be1f92d7a8a8ebacd257a"}
Jul 29 18:34:37.719085 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.719056    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66" event={"ID":"b4e01106-0034-44ad-99dd-f438ef1204c2","Type":"ContainerStarted","Data":"dc0b08eca2ba7c4a14968d266b37b7563ebfc74b76d867efb65547dffb138699"}
Jul 29 18:34:37.729775 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.729749    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2qm5s" event={"ID":"990500cb-5912-4252-b507-f499bae20618","Type":"ContainerStarted","Data":"e8cd4a85a40655af63058862e8dbcbe81050e46616c95a14aa95919fa7851ea5"}
Jul 29 18:34:37.731041 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.731019    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2qlfn" event={"ID":"c98ed70e-1aab-43f0-b57d-192dac28efd6","Type":"ContainerStarted","Data":"ce44f7079acbb2a1d112ccfab04028931b66f770e8221f040b524bf02707eb91"}
Jul 29 18:34:37.733794 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.733773    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfw8j" event={"ID":"396aaa29-be70-4f0b-bd94-a3cc455a5cd2","Type":"ContainerStarted","Data":"4a81373a0aa0467cbf556bdf39fb4504425d73cc80431c9941fbb7f48b0d5582"}
Jul 29 18:34:37.739262 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.739238    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tffp8" event={"ID":"a4ea3227-0889-453b-b4af-696d070a0174","Type":"ContainerStarted","Data":"7db6d112cccfe41473556cf722d38eec5c85f3af51044a73ad27d71f722cceb8"}
Jul 29 18:34:37.743342 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.743318    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x6xsd" event={"ID":"51e71484-cd8e-4585-b7a3-c14fb8374d74","Type":"ContainerStarted","Data":"09c0ce1a92bfca4b0072899a0f60fa3370a6b46775c6bf96109ea3214e6f64a0"}
Jul 29 18:34:37.748882 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.748772    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd" event={"ID":"1072bb12-b457-4240-abf9-fbba51caa986","Type":"ContainerStarted","Data":"74514c0cb314803739e59f0e3b8e95c59088790de7102e74c202452de1fd9e23"}
Jul 29 18:34:37.753120 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.753091    2560 generic.go:356] "Generic (PLEG): container finished" podID="5cda4b95d9841dcfa4719835180ed2ce" containerID="4d35e0d95c99d6b8acef4c9035221ba3621bfd246254c7fcdfb09f8f143b6996" exitCode=0
Jul 29 18:34:37.753229 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.753131    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal" event={"ID":"5cda4b95d9841dcfa4719835180ed2ce","Type":"ContainerDied","Data":"4d35e0d95c99d6b8acef4c9035221ba3621bfd246254c7fcdfb09f8f143b6996"}
Jul 29 18:34:37.862476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:37.862433    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 18:34:38.075999 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:38.075960    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:38.076176 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:38.076130    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:38.076266 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:38.076210    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret podName:244afbce-35b6-4432-a874-d7081b20c82e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:40.076175965 +0000 UTC m=+5.970176360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret") pod "global-pull-secret-syncer-rrsh7" (UID: "244afbce-35b6-4432-a874-d7081b20c82e") : object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:38.176758 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:38.176705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs\") pod \"network-metrics-daemon-2jvrj\" (UID: \"7154111a-1f26-4f41-8ba6-94617dcca1b3\") " pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:38.176919 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:38.176782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdq9w\" (UniqueName: \"kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w\") pod \"network-check-target-44mt4\" (UID: \"7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e\") " pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:38.176990 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:38.176915    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 18:34:38.176990 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:38.176932    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 18:34:38.176990 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:38.176945    2560 projected.go:196] Error preparing data for projected volume kube-api-access-bdq9w for pod openshift-network-diagnostics/network-check-target-44mt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:38.177133 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:38.177001    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w podName:7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:40.17698357 +0000 UTC m=+6.070983952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bdq9w" (UniqueName: "kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w") pod "network-check-target-44mt4" (UID: "7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:38.177435 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:38.177417    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:38.177520 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:38.177471    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs podName:7154111a-1f26-4f41-8ba6-94617dcca1b3 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:40.17745657 +0000 UTC m=+6.071456953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs") pod "network-metrics-daemon-2jvrj" (UID: "7154111a-1f26-4f41-8ba6-94617dcca1b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:38.494607 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:38.494531    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 18:29:36 +0000 UTC" deadline="2028-04-08 23:23:45.319187412 +0000 UTC"
Jul 29 18:34:38.494607 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:38.494573    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14860h49m6.824620176s"
Jul 29 18:34:38.688217 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:38.685101    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:38.688217 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:38.685262    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:34:38.688217 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:38.685643    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:38.688217 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:38.685743    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:34:38.779421 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:38.778390    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal" event={"ID":"5cda4b95d9841dcfa4719835180ed2ce","Type":"ContainerStarted","Data":"957ad1e18fe5144229be5c35e0d5398813295a51b1cebbca6d3c05b17c223607"}
Jul 29 18:34:39.681836 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:39.681803    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:39.682311 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:39.681933    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:34:40.095382 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:40.095350    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:40.095556 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:40.095514    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:40.095640 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:40.095577    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret podName:244afbce-35b6-4432-a874-d7081b20c82e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:44.095558234 +0000 UTC m=+9.989558696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret") pod "global-pull-secret-syncer-rrsh7" (UID: "244afbce-35b6-4432-a874-d7081b20c82e") : object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:40.195811 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:40.195762    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs\") pod \"network-metrics-daemon-2jvrj\" (UID: \"7154111a-1f26-4f41-8ba6-94617dcca1b3\") " pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:40.195980 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:40.195838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdq9w\" (UniqueName: \"kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w\") pod \"network-check-target-44mt4\" (UID: \"7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e\") " pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:40.196041 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:40.195979    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 18:34:40.196041 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:40.195999    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 18:34:40.196041 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:40.196012    2560 projected.go:196] Error preparing data for projected volume kube-api-access-bdq9w for pod openshift-network-diagnostics/network-check-target-44mt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:40.196233 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:40.196067    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w podName:7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:44.196049898 +0000 UTC m=+10.090050282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bdq9w" (UniqueName: "kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w") pod "network-check-target-44mt4" (UID: "7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:40.196509 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:40.196489    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:40.196617 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:40.196550    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs podName:7154111a-1f26-4f41-8ba6-94617dcca1b3 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:44.196534916 +0000 UTC m=+10.090535300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs") pod "network-metrics-daemon-2jvrj" (UID: "7154111a-1f26-4f41-8ba6-94617dcca1b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:40.682513 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:40.682478    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:40.683010 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:40.682619    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:34:40.683010 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:40.682479    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:40.683124 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:40.683082    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:34:41.681549 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:41.681509    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:41.681729 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:41.681653    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:34:42.682165 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:42.682068    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:42.682165 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:42.682084    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:42.682759 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:42.682207    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:34:42.682759 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:42.682278    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:34:43.681800 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:43.681770    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:43.681982 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:43.681878    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:34:44.128469 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:44.128358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:44.128956 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:44.128528    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:44.128956 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:44.128616    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret podName:244afbce-35b6-4432-a874-d7081b20c82e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:52.128591513 +0000 UTC m=+18.022591898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret") pod "global-pull-secret-syncer-rrsh7" (UID: "244afbce-35b6-4432-a874-d7081b20c82e") : object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:44.229650 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:44.229576    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdq9w\" (UniqueName: \"kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w\") pod \"network-check-target-44mt4\" (UID: \"7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e\") " pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:44.229825 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:44.229657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs\") pod \"network-metrics-daemon-2jvrj\" (UID: \"7154111a-1f26-4f41-8ba6-94617dcca1b3\") " pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:44.229825 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:44.229807    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:44.229944 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:44.229874    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs podName:7154111a-1f26-4f41-8ba6-94617dcca1b3 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:52.229854828 +0000 UTC m=+18.123855207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs") pod "network-metrics-daemon-2jvrj" (UID: "7154111a-1f26-4f41-8ba6-94617dcca1b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:44.230286 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:44.230146    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 18:34:44.230286 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:44.230176    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 18:34:44.230286 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:44.230191    2560 projected.go:196] Error preparing data for projected volume kube-api-access-bdq9w for pod openshift-network-diagnostics/network-check-target-44mt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:44.230286 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:44.230262    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w podName:7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:52.230244991 +0000 UTC m=+18.124245393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bdq9w" (UniqueName: "kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w") pod "network-check-target-44mt4" (UID: "7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:44.682782 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:44.682714    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:44.684415 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:44.683847    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:34:44.684415 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:44.684269    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:44.684415 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:44.684368    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:34:45.682054 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:45.682023    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:45.682505 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:45.682141    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:34:46.682359 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:46.682320    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:46.682851 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:46.682320    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:46.682851 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:46.682474    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:34:46.682851 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:46.682498    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:34:47.682243 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:47.682213    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:47.682393 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:47.682312    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:34:48.682020 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:48.681982    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:48.682175 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:48.682121    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:34:48.682175 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:48.682154    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:48.682275 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:48.682229    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:34:49.682505 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:49.682471    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:49.683034 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:49.682593    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:34:50.682136 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:50.681991    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:50.682136 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:50.682131    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:34:50.682395 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:50.682172    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:50.682395 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:50.682306    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:34:51.681465 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:51.681425    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:51.681885 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:51.681540    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:34:52.192135 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:52.192093    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:52.192427 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:52.192266    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:52.192427 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:52.192344    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret podName:244afbce-35b6-4432-a874-d7081b20c82e nodeName:}" failed. No retries permitted until 2026-07-29 18:35:08.192320876 +0000 UTC m=+34.086321272 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret") pod "global-pull-secret-syncer-rrsh7" (UID: "244afbce-35b6-4432-a874-d7081b20c82e") : object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:52.293348 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:52.293311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdq9w\" (UniqueName: \"kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w\") pod \"network-check-target-44mt4\" (UID: \"7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e\") " pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:52.293527 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:52.293383    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs\") pod \"network-metrics-daemon-2jvrj\" (UID: \"7154111a-1f26-4f41-8ba6-94617dcca1b3\") " pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:52.293527 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:52.293485    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:52.293639 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:52.293550    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs podName:7154111a-1f26-4f41-8ba6-94617dcca1b3 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:08.293531963 +0000 UTC m=+34.187532361 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs") pod "network-metrics-daemon-2jvrj" (UID: "7154111a-1f26-4f41-8ba6-94617dcca1b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:52.293639 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:52.293486    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 18:34:52.293639 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:52.293617    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 18:34:52.293639 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:52.293633    2560 projected.go:196] Error preparing data for projected volume kube-api-access-bdq9w for pod openshift-network-diagnostics/network-check-target-44mt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:52.293829 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:52.293682    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w podName:7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e nodeName:}" failed. No retries permitted until 2026-07-29 18:35:08.29366924 +0000 UTC m=+34.187669620 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bdq9w" (UniqueName: "kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w") pod "network-check-target-44mt4" (UID: "7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:52.682426 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:52.682390    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:52.682914 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:52.682402    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:52.682914 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:52.682524    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:34:52.682914 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:52.682610    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:34:53.682294 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:53.682257    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:53.682482 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:53.682375    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:34:54.683083 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:54.683030    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:54.683528 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:54.683142    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:34:54.683528 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:54.683233    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:54.683528 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:54.683354    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:34:55.681721 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:55.681688    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:55.681882 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:55.681780    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:34:56.682264 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.682077    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:56.683077 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:56.682356    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:34:56.683077 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.682801    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:56.683077 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:56.682905    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:34:56.819387 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.819353    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 18:34:56.819724 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.819700    2560 generic.go:356] "Generic (PLEG): container finished" podID="1072bb12-b457-4240-abf9-fbba51caa986" containerID="a1ea51f1eee4ec694deb99a2f4bc14b95f26e206d5dbbc39bacbfc384d9c2d36" exitCode=1
Jul 29 18:34:56.819821 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.819772    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd" event={"ID":"1072bb12-b457-4240-abf9-fbba51caa986","Type":"ContainerStarted","Data":"bc2d0b6b7e2cfe24b4aad108e7fa81e0187354f334c84b7a9c9ad95be5265c0b"}
Jul 29 18:34:56.819878 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.819861    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd" event={"ID":"1072bb12-b457-4240-abf9-fbba51caa986","Type":"ContainerStarted","Data":"6fde1d012cf13a4811b2bb9386b5279e5b044499e2143ca242e01001e9891306"}
Jul 29 18:34:56.819930 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.819916    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd" event={"ID":"1072bb12-b457-4240-abf9-fbba51caa986","Type":"ContainerStarted","Data":"720e7fff3cd9f320346bb81b52de6abfcc9fe7fd0a3474fab63bdf77da1b9c15"}
Jul 29 18:34:56.819976 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.819964    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd" event={"ID":"1072bb12-b457-4240-abf9-fbba51caa986","Type":"ContainerStarted","Data":"68baec312c124e288113f84f4537e5c36c5a5996e89e37bc7c732702cbcfc6e8"}
Jul 29 18:34:56.820025 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.820005    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd" event={"ID":"1072bb12-b457-4240-abf9-fbba51caa986","Type":"ContainerDied","Data":"a1ea51f1eee4ec694deb99a2f4bc14b95f26e206d5dbbc39bacbfc384d9c2d36"}
Jul 29 18:34:56.820098 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.820050    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd" event={"ID":"1072bb12-b457-4240-abf9-fbba51caa986","Type":"ContainerStarted","Data":"510d126cb87e0b53f7deec088eaaaec4d807ef8b198c0a79267755a8e508e628"}
Jul 29 18:34:56.821303 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.821277    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-52.ec2.internal" event={"ID":"25ae268d780129888ee17e93df9f46c1","Type":"ContainerStarted","Data":"75b70c4b9f760369e39ac88b3785d70c844edf3e084ed9f302e9ea1c8dd78779"}
Jul 29 18:34:56.822683 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.822665    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r7k7j" event={"ID":"fb20b3c0-7818-479d-8fe3-3c5700547014","Type":"ContainerStarted","Data":"4a8f4d19ed24f23618153cfa11fa38368399205ec564d8b5a5a45be4eccb91ed"}
Jul 29 18:34:56.824370 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.824344    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6d6sh" event={"ID":"2c4effb3-c122-47d9-88ad-557670f8248d","Type":"ContainerStarted","Data":"879f92ab32a6971bf4ab6ccbea87106f62e1bf8c6ee916d4b082824523059064"}
Jul 29 18:34:56.825887 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.825870    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66" event={"ID":"b4e01106-0034-44ad-99dd-f438ef1204c2","Type":"ContainerStarted","Data":"19f01f4db8c7ffbbcde8239bf8f1e81a71bd2e2fd802f3db38fe5d5b08bd6a57"}
Jul 29 18:34:56.827253 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.827230    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2qm5s" event={"ID":"990500cb-5912-4252-b507-f499bae20618","Type":"ContainerStarted","Data":"7d0ef632f857b6a4482f3db7966e81112dae0cf01a414b7a88f60f8223115b05"}
Jul 29 18:34:56.828573 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.828552    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2qlfn" event={"ID":"c98ed70e-1aab-43f0-b57d-192dac28efd6","Type":"ContainerStarted","Data":"3fcf1741481a243a6e4d99f5c465bbe674520ab84ccb74c46bf2d6ad6c710a38"}
Jul 29 18:34:56.830071 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.830048    2560 generic.go:356] "Generic (PLEG): container finished" podID="396aaa29-be70-4f0b-bd94-a3cc455a5cd2" containerID="2ee5a7067ae692e35e579fa4782f5abf3fd812dcbd0a4dd67661a3ac01ba97ef" exitCode=0
Jul 29 18:34:56.830175 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.830083    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfw8j" event={"ID":"396aaa29-be70-4f0b-bd94-a3cc455a5cd2","Type":"ContainerDied","Data":"2ee5a7067ae692e35e579fa4782f5abf3fd812dcbd0a4dd67661a3ac01ba97ef"}
Jul 29 18:34:56.831827 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.831471    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tffp8" event={"ID":"a4ea3227-0889-453b-b4af-696d070a0174","Type":"ContainerStarted","Data":"bc3c9ee9080f0d84252d088fa095524a568267fc7e7b1e8c831c3b1a31d344ca"}
Jul 29 18:34:56.835175 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.835113    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-52.ec2.internal" podStartSLOduration=21.835097185 podStartE2EDuration="21.835097185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:38.79616084 +0000 UTC m=+4.690161243" watchObservedRunningTime="2026-07-29 18:34:56.835097185 +0000 UTC m=+22.729097589"
Jul 29 18:34:56.835663 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.835633    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-52.ec2.internal" podStartSLOduration=21.835624915 podStartE2EDuration="21.835624915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:56.834915882 +0000 UTC m=+22.728916330" watchObservedRunningTime="2026-07-29 18:34:56.835624915 +0000 UTC m=+22.729625318"
Jul 29 18:34:56.874533 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.874471    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r7k7j" podStartSLOduration=3.195426949 podStartE2EDuration="22.8744467s" totalImagesPullingTime="19.679019751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:56.873515382 +0000 UTC m=+22.767515786" watchObservedRunningTime="2026-07-29 18:34:56.8744467 +0000 UTC m=+22.768447102"
Jul 29 18:34:56.887847 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.887794    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2qlfn" podStartSLOduration=3.716358875 podStartE2EDuration="22.887773524s" totalImagesPullingTime="19.171414649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:56.887745596 +0000 UTC m=+22.781746028" watchObservedRunningTime="2026-07-29 18:34:56.887773524 +0000 UTC m=+22.781773925"
Jul 29 18:34:56.907568 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.907521    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tffp8" podStartSLOduration=3.743808438 podStartE2EDuration="22.907506915s" totalImagesPullingTime="19.163698477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:56.90731852 +0000 UTC m=+22.801318949" watchObservedRunningTime="2026-07-29 18:34:56.907506915 +0000 UTC m=+22.801507315"
Jul 29 18:34:56.922698 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.922646    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2qm5s" podStartSLOduration=3.747627735 podStartE2EDuration="22.922627102s" totalImagesPullingTime="19.174999367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:56.921636635 +0000 UTC m=+22.815637037" watchObservedRunningTime="2026-07-29 18:34:56.922627102 +0000 UTC m=+22.816627504"
Jul 29 18:34:56.935783 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:56.935741    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6d6sh" podStartSLOduration=11.950051366 podStartE2EDuration="22.935728052s" totalImagesPullingTime="10.985676686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:56.935020451 +0000 UTC m=+22.829020853" watchObservedRunningTime="2026-07-29 18:34:56.935728052 +0000 UTC m=+22.829728455"
Jul 29 18:34:57.398548 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:57.398519    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 18:34:57.526193 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:57.526023    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T18:34:57.398540715Z","UUID":"40ca571e-cb49-4398-b06a-bbb77696da51","Handler":null,"Name":"","Endpoint":""}
Jul 29 18:34:57.527808 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:57.527785    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 18:34:57.527952 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:57.527826    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 18:34:57.681816 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:57.681790    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:57.681998 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:57.681914    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:34:57.836292 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:57.836254    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66" event={"ID":"b4e01106-0034-44ad-99dd-f438ef1204c2","Type":"ContainerStarted","Data":"10262d4a9ef4bb9b6ca2aa360432c1facdb507a4a2b220095c9d616ac2be1365"}
Jul 29 18:34:57.840084 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:57.839984    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x6xsd" event={"ID":"51e71484-cd8e-4585-b7a3-c14fb8374d74","Type":"ContainerStarted","Data":"5af3644fd9dba1e3430e042f14c2c860ef355c6b921df2a33f518a03066bc5db"}
Jul 29 18:34:58.051679 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:58.051641    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6d6sh"
Jul 29 18:34:58.052558 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:58.052524    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6d6sh"
Jul 29 18:34:58.069087 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:58.069031    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x6xsd" podStartSLOduration=4.905168991 podStartE2EDuration="24.069015078s" totalImagesPullingTime="19.163846087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:57.857592628 +0000 UTC m=+23.751593030" watchObservedRunningTime="2026-07-29 18:34:58.069015078 +0000 UTC m=+23.963015480"
Jul 29 18:34:58.682025 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:58.681993    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:34:58.682297 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:58.682006    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:34:58.682297 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:58.682121    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:34:58.682297 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:58.682225    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:34:58.844637 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:58.844600    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66" event={"ID":"b4e01106-0034-44ad-99dd-f438ef1204c2","Type":"ContainerStarted","Data":"5ebb842057b65bcf11808515eab7a9d5b5917bffa29c145ebab5e6eb9b89a485"}
Jul 29 18:34:58.845513 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:58.845057    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6d6sh"
Jul 29 18:34:58.845513 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:58.845451    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6d6sh"
Jul 29 18:34:58.863061 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:58.863012    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjp66" podStartSLOduration=3.605089344 podStartE2EDuration="24.862997539s" totalImagesPullingTime="21.257908195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:58.862544894 +0000 UTC m=+24.756545321" watchObservedRunningTime="2026-07-29 18:34:58.862997539 +0000 UTC m=+24.756997939"
Jul 29 18:34:59.682631 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:59.682447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:34:59.682913 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:34:59.682738    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:34:59.850064 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:59.850037    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 18:34:59.850548 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:34:59.850521    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd" event={"ID":"1072bb12-b457-4240-abf9-fbba51caa986","Type":"ContainerStarted","Data":"c2eeb415ec6d9caf50f4c7b5137e0813ee5226bcf67e1e293fc1db7092161f29"}
Jul 29 18:35:00.682305 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:00.682267    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:35:00.682533 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:00.682390    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:35:00.682533 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:00.682448    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:35:00.682649 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:00.682570    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:35:01.682106 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:01.681937    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:35:01.682901 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:01.682179    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:35:01.856307 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:01.856275    2560 generic.go:356] "Generic (PLEG): container finished" podID="396aaa29-be70-4f0b-bd94-a3cc455a5cd2" containerID="7c6be8de91f5350a04b1decb58989a775b08aa6f121d9dd225a1721e195ac2ab" exitCode=0
Jul 29 18:35:01.856480 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:01.856360    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfw8j" event={"ID":"396aaa29-be70-4f0b-bd94-a3cc455a5cd2","Type":"ContainerDied","Data":"7c6be8de91f5350a04b1decb58989a775b08aa6f121d9dd225a1721e195ac2ab"}
Jul 29 18:35:01.860151 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:01.860130    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 18:35:01.860527 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:01.860505    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd" event={"ID":"1072bb12-b457-4240-abf9-fbba51caa986","Type":"ContainerStarted","Data":"10c7ffb655f1139207a5bb4a25342f8f20d78890ce52154f266ce3e77af9ef68"}
Jul 29 18:35:01.860749 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:01.860736    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:35:01.860796 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:01.860785    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:35:01.860883 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:01.860867    2560 scope.go:117] "RemoveContainer" containerID="a1ea51f1eee4ec694deb99a2f4bc14b95f26e206d5dbbc39bacbfc384d9c2d36"
Jul 29 18:35:01.878025 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:01.878009    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:35:02.684058 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:02.683964    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:35:02.684478 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:02.684086    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:35:02.684478 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:02.684122    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:35:02.684478 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:02.684269    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:35:02.866634 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:02.866611    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 18:35:02.867020 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:02.866996    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd" event={"ID":"1072bb12-b457-4240-abf9-fbba51caa986","Type":"ContainerStarted","Data":"9d2a7aa05e603bffd13e0e63c545d692d75be0f3aadfb5719f71d4578049777b"}
Jul 29 18:35:02.867256 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:02.867230    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:35:02.882613 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:02.882593    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:35:02.900295 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:02.900241    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd" podStartSLOduration=9.707013726 podStartE2EDuration="28.900220788s" totalImagesPullingTime="19.193207062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:02.898366504 +0000 UTC m=+28.792366906" watchObservedRunningTime="2026-07-29 18:35:02.900220788 +0000 UTC m=+28.794221183"
Jul 29 18:35:03.326390 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:03.326352    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rrsh7"]
Jul 29 18:35:03.326582 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:03.326473    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:35:03.326582 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:03.326553    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:35:03.329669 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:03.329599    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2jvrj"]
Jul 29 18:35:03.329796 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:03.329682    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:35:03.329796 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:03.329766    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:35:03.330592 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:03.330572    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-44mt4"]
Jul 29 18:35:03.330689 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:03.330635    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:35:03.330749 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:03.330696    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:35:03.871601 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:03.871415    2560 generic.go:356] "Generic (PLEG): container finished" podID="396aaa29-be70-4f0b-bd94-a3cc455a5cd2" containerID="0f6851c6ec06bd26a05c34cbc5d2c8410fcee3b36f6829c9a1b380cab8bc3da4" exitCode=0
Jul 29 18:35:03.871601 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:03.871498    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfw8j" event={"ID":"396aaa29-be70-4f0b-bd94-a3cc455a5cd2","Type":"ContainerDied","Data":"0f6851c6ec06bd26a05c34cbc5d2c8410fcee3b36f6829c9a1b380cab8bc3da4"}
Jul 29 18:35:04.682465 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:04.682434    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:35:04.682617 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:04.682518    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:35:05.681747 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:05.681715    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:35:05.682283 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:05.681835    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:35:05.682283 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:05.681895    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:35:05.682283 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:05.682005    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:35:06.681624 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:06.681588    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:35:06.681787 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:06.681698    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:35:06.880538 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:06.880505    2560 generic.go:356] "Generic (PLEG): container finished" podID="396aaa29-be70-4f0b-bd94-a3cc455a5cd2" containerID="e161c4a88e3f502fc62f360012f7d4c655655630ae2af0a5c63d81522d1704f0" exitCode=0
Jul 29 18:35:06.880695 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:06.880567    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfw8j" event={"ID":"396aaa29-be70-4f0b-bd94-a3cc455a5cd2","Type":"ContainerDied","Data":"e161c4a88e3f502fc62f360012f7d4c655655630ae2af0a5c63d81522d1704f0"}
Jul 29 18:35:07.681823 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:07.681744    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:35:07.681823 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:07.681779    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:35:07.682279 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:07.681900    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrsh7" podUID="244afbce-35b6-4432-a874-d7081b20c82e"
Jul 29 18:35:07.682279 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:07.682090    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jvrj" podUID="7154111a-1f26-4f41-8ba6-94617dcca1b3"
Jul 29 18:35:08.208860 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:08.208820    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:35:08.209019 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:08.208976    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 18:35:08.209067 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:08.209043    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret podName:244afbce-35b6-4432-a874-d7081b20c82e nodeName:}" failed. No retries permitted until 2026-07-29 18:35:40.209027456 +0000 UTC m=+66.103027840 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret") pod "global-pull-secret-syncer-rrsh7" (UID: "244afbce-35b6-4432-a874-d7081b20c82e") : object "kube-system"/"original-pull-secret" not registered
Jul 29 18:35:08.309611 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:08.309574    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs\") pod \"network-metrics-daemon-2jvrj\" (UID: \"7154111a-1f26-4f41-8ba6-94617dcca1b3\") " pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:35:08.309800 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:08.309726    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:35:08.309800 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:08.309752    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdq9w\" (UniqueName: \"kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w\") pod \"network-check-target-44mt4\" (UID: \"7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e\") " pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:35:08.309800 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:08.309786    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs podName:7154111a-1f26-4f41-8ba6-94617dcca1b3 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:40.309770953 +0000 UTC m=+66.203771331 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs") pod "network-metrics-daemon-2jvrj" (UID: "7154111a-1f26-4f41-8ba6-94617dcca1b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:35:08.309977 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:08.309861    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 18:35:08.309977 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:08.309878    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 18:35:08.309977 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:08.309886    2560 projected.go:196] Error preparing data for projected volume kube-api-access-bdq9w for pod openshift-network-diagnostics/network-check-target-44mt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:35:08.309977 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:08.309934    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w podName:7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e nodeName:}" failed. No retries permitted until 2026-07-29 18:35:40.309921014 +0000 UTC m=+66.203921396 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bdq9w" (UniqueName: "kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w") pod "network-check-target-44mt4" (UID: "7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:35:08.682346 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:08.682313    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:35:08.682907 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:08.682446    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mt4" podUID="7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e"
Jul 29 18:35:09.472208 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.472165    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-52.ec2.internal" event="NodeReady"
Jul 29 18:35:09.472354 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.472339    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 18:35:09.521246 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.521208    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w6dzw"]
Jul 29 18:35:09.523766 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.523745    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:09.525597 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.525571    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dw6dv"]
Jul 29 18:35:09.526652 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.526578    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hgmvm\""
Jul 29 18:35:09.526800 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.526721    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 18:35:09.526884 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.526816    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 18:35:09.527317 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.527299    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dw6dv"
Jul 29 18:35:09.529878 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.529856    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 18:35:09.530012 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.529996    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 18:35:09.530116 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.530090    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q2mp4\""
Jul 29 18:35:09.530116 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.530101    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 18:35:09.538759 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.538739    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w6dzw"]
Jul 29 18:35:09.539678 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.539497    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dw6dv"]
Jul 29 18:35:09.621672 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.621633    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5830c3a8-6329-4806-b3f0-1cfb9141886d-cert\") pod \"ingress-canary-dw6dv\" (UID: \"5830c3a8-6329-4806-b3f0-1cfb9141886d\") " pod="openshift-ingress-canary/ingress-canary-dw6dv"
Jul 29 18:35:09.621854 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.621684    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3218c46-a240-4e04-b5a6-1aa3b20524ce-tmp-dir\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:09.621854 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.621780    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp5zt\" (UniqueName: \"kubernetes.io/projected/5830c3a8-6329-4806-b3f0-1cfb9141886d-kube-api-access-jp5zt\") pod \"ingress-canary-dw6dv\" (UID: \"5830c3a8-6329-4806-b3f0-1cfb9141886d\") " pod="openshift-ingress-canary/ingress-canary-dw6dv"
Jul 29 18:35:09.621854 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.621815    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5gzx\" (UniqueName: \"kubernetes.io/projected/e3218c46-a240-4e04-b5a6-1aa3b20524ce-kube-api-access-r5gzx\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:09.621997 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.621885    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3218c46-a240-4e04-b5a6-1aa3b20524ce-config-volume\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:09.621997 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.621912    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:09.681895 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.681856    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:35:09.681895 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.681885    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:35:09.684519 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.684483    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 18:35:09.684921 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.684547    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 18:35:09.684921 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.684632    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g7g2r\""
Jul 29 18:35:09.723219 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.723129    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3218c46-a240-4e04-b5a6-1aa3b20524ce-config-volume\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:09.723219 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.723167    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:09.723406 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.723240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5830c3a8-6329-4806-b3f0-1cfb9141886d-cert\") pod \"ingress-canary-dw6dv\" (UID: \"5830c3a8-6329-4806-b3f0-1cfb9141886d\") " pod="openshift-ingress-canary/ingress-canary-dw6dv"
Jul 29 18:35:09.723406 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.723267    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3218c46-a240-4e04-b5a6-1aa3b20524ce-tmp-dir\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:09.723406 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.723303    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5zt\" (UniqueName: \"kubernetes.io/projected/5830c3a8-6329-4806-b3f0-1cfb9141886d-kube-api-access-jp5zt\") pod \"ingress-canary-dw6dv\" (UID: \"5830c3a8-6329-4806-b3f0-1cfb9141886d\") " pod="openshift-ingress-canary/ingress-canary-dw6dv"
Jul 29 18:35:09.723406 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.723319    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5gzx\" (UniqueName: \"kubernetes.io/projected/e3218c46-a240-4e04-b5a6-1aa3b20524ce-kube-api-access-r5gzx\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:09.723406 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:09.723390    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 18:35:09.723637 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:09.723468    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls podName:e3218c46-a240-4e04-b5a6-1aa3b20524ce nodeName:}" failed. No retries permitted until 2026-07-29 18:35:10.223446276 +0000 UTC m=+36.117446678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls") pod "dns-default-w6dzw" (UID: "e3218c46-a240-4e04-b5a6-1aa3b20524ce") : secret "dns-default-metrics-tls" not found
Jul 29 18:35:09.723762 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.723742    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3218c46-a240-4e04-b5a6-1aa3b20524ce-config-volume\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:09.723846 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.723770    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3218c46-a240-4e04-b5a6-1aa3b20524ce-tmp-dir\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:09.728281 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.728259    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5830c3a8-6329-4806-b3f0-1cfb9141886d-cert\") pod \"ingress-canary-dw6dv\" (UID: \"5830c3a8-6329-4806-b3f0-1cfb9141886d\") " pod="openshift-ingress-canary/ingress-canary-dw6dv"
Jul 29 18:35:09.733984 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.733937    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5gzx\" (UniqueName: \"kubernetes.io/projected/e3218c46-a240-4e04-b5a6-1aa3b20524ce-kube-api-access-r5gzx\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:09.734276 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.734254    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5zt\" (UniqueName: \"kubernetes.io/projected/5830c3a8-6329-4806-b3f0-1cfb9141886d-kube-api-access-jp5zt\") pod \"ingress-canary-dw6dv\" (UID: \"5830c3a8-6329-4806-b3f0-1cfb9141886d\") " pod="openshift-ingress-canary/ingress-canary-dw6dv"
Jul 29 18:35:09.842428 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:09.842396    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dw6dv"
Jul 29 18:35:10.227179 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:10.227139    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:10.227366 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:10.227303    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 18:35:10.227408 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:10.227386    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls podName:e3218c46-a240-4e04-b5a6-1aa3b20524ce nodeName:}" failed. No retries permitted until 2026-07-29 18:35:11.227369037 +0000 UTC m=+37.121369415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls") pod "dns-default-w6dzw" (UID: "e3218c46-a240-4e04-b5a6-1aa3b20524ce") : secret "dns-default-metrics-tls" not found
Jul 29 18:35:10.681913 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:10.681869    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:35:10.684399 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:10.684372    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 18:35:10.684545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:10.684475    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b24qk\""
Jul 29 18:35:10.684893 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:10.684547    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 18:35:11.059570 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.059545    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dw6dv"]
Jul 29 18:35:11.063343 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:35:11.063321    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5830c3a8_6329_4806_b3f0_1cfb9141886d.slice/crio-93b37491eab35afa87edb06a6f704c616fc100ec92c61909bf09bd8756705997 WatchSource:0}: Error finding container 93b37491eab35afa87edb06a6f704c616fc100ec92c61909bf09bd8756705997: Status 404 returned error can't find the container with id 93b37491eab35afa87edb06a6f704c616fc100ec92c61909bf09bd8756705997
Jul 29 18:35:11.235990 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.235907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:11.236123 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:11.236076    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 18:35:11.236157 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:11.236147    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls podName:e3218c46-a240-4e04-b5a6-1aa3b20524ce nodeName:}" failed. No retries permitted until 2026-07-29 18:35:13.236129875 +0000 UTC m=+39.130130253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls") pod "dns-default-w6dzw" (UID: "e3218c46-a240-4e04-b5a6-1aa3b20524ce") : secret "dns-default-metrics-tls" not found
Jul 29 18:35:11.418660 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.418622    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck"]
Jul 29 18:35:11.420392 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.420376    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck"
Jul 29 18:35:11.422803 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.422775    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 29 18:35:11.423057 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.423040    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 29 18:35:11.423111 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.423064    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 29 18:35:11.423745 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.423728    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 29 18:35:11.423824 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.423727    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-57fsx\""
Jul 29 18:35:11.428961 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.428942    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck"]
Jul 29 18:35:11.538592 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.538499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8e92808-8685-4cd5-b575-e22d16dc483a-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck\" (UID: \"a8e92808-8685-4cd5-b575-e22d16dc483a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck"
Jul 29 18:35:11.538592 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.538566    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmrx8\" (UniqueName: \"kubernetes.io/projected/a8e92808-8685-4cd5-b575-e22d16dc483a-kube-api-access-qmrx8\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck\" (UID: \"a8e92808-8685-4cd5-b575-e22d16dc483a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck"
Jul 29 18:35:11.538809 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.538709    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8e92808-8685-4cd5-b575-e22d16dc483a-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck\" (UID: \"a8e92808-8685-4cd5-b575-e22d16dc483a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck"
Jul 29 18:35:11.640029 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.639994    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8e92808-8685-4cd5-b575-e22d16dc483a-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck\" (UID: \"a8e92808-8685-4cd5-b575-e22d16dc483a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck"
Jul 29 18:35:11.640226 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.640041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmrx8\" (UniqueName: \"kubernetes.io/projected/a8e92808-8685-4cd5-b575-e22d16dc483a-kube-api-access-qmrx8\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck\" (UID: \"a8e92808-8685-4cd5-b575-e22d16dc483a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck"
Jul 29 18:35:11.640226 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.640087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8e92808-8685-4cd5-b575-e22d16dc483a-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck\" (UID: \"a8e92808-8685-4cd5-b575-e22d16dc483a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck"
Jul 29 18:35:11.640740 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.640717    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8e92808-8685-4cd5-b575-e22d16dc483a-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck\" (UID: \"a8e92808-8685-4cd5-b575-e22d16dc483a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck"
Jul 29 18:35:11.643800 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.643771    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8e92808-8685-4cd5-b575-e22d16dc483a-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck\" (UID: \"a8e92808-8685-4cd5-b575-e22d16dc483a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck"
Jul 29 18:35:11.648969 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.648937    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmrx8\" (UniqueName: \"kubernetes.io/projected/a8e92808-8685-4cd5-b575-e22d16dc483a-kube-api-access-qmrx8\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck\" (UID: \"a8e92808-8685-4cd5-b575-e22d16dc483a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck"
Jul 29 18:35:11.661923 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.661889    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:11.729564 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.729527    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck"
Jul 29 18:35:11.782591 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.782393    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:11.868411 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.868302    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck"]
Jul 29 18:35:11.894348 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.894311    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dw6dv" event={"ID":"5830c3a8-6329-4806-b3f0-1cfb9141886d","Type":"ContainerStarted","Data":"93b37491eab35afa87edb06a6f704c616fc100ec92c61909bf09bd8756705997"}
Jul 29 18:35:11.897898 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.897839    2560 generic.go:356] "Generic (PLEG): container finished" podID="396aaa29-be70-4f0b-bd94-a3cc455a5cd2" containerID="670afe71073cbd8097e96d6375b8283b1d57288d996f500a08dfca28aec8564a" exitCode=0
Jul 29 18:35:11.898000 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.897915    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfw8j" event={"ID":"396aaa29-be70-4f0b-bd94-a3cc455a5cd2","Type":"ContainerDied","Data":"670afe71073cbd8097e96d6375b8283b1d57288d996f500a08dfca28aec8564a"}
Jul 29 18:35:11.899188 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:11.899165    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck" event={"ID":"a8e92808-8685-4cd5-b575-e22d16dc483a","Type":"ContainerStarted","Data":"7b0f05c29cf6ab5ca20b85a942814cde4a6f81c5a2e1d9bfe33b4c35efdf8512"}
Jul 29 18:35:12.905640 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:12.905603    2560 generic.go:356] "Generic (PLEG): container finished" podID="396aaa29-be70-4f0b-bd94-a3cc455a5cd2" containerID="2c694612ec314f8bf0bd39f48e6e30ee8a5f9f8c09bd254d3f1c7613d731f9df" exitCode=0
Jul 29 18:35:12.906301 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:12.905659    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfw8j" event={"ID":"396aaa29-be70-4f0b-bd94-a3cc455a5cd2","Type":"ContainerDied","Data":"2c694612ec314f8bf0bd39f48e6e30ee8a5f9f8c09bd254d3f1c7613d731f9df"}
Jul 29 18:35:13.142645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.142602    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:13.206778 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.206461    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:13.254859 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.254823    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:13.255043 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:13.255015    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 18:35:13.255179 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:13.255117    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls podName:e3218c46-a240-4e04-b5a6-1aa3b20524ce nodeName:}" failed. No retries permitted until 2026-07-29 18:35:17.255094762 +0000 UTC m=+43.149095144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls") pod "dns-default-w6dzw" (UID: "e3218c46-a240-4e04-b5a6-1aa3b20524ce") : secret "dns-default-metrics-tls" not found
Jul 29 18:35:13.343274 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.343241    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:13.446783 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.446642    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:13.545736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.545688    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:13.702485 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.702410    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bqkx8"]
Jul 29 18:35:13.718529 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.718488    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bqkx8"]
Jul 29 18:35:13.718692 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.718616    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:13.721412 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.721219    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 18:35:13.721412 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.721254    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 18:35:13.721412 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.721341    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 18:35:13.721412 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.721351    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8px9m\""
Jul 29 18:35:13.721675 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.721656    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 18:35:13.738835 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.738806    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:13.799589 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.799527    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:13.860038 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.860005    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40e67943-b279-4be0-bf71-976537ed7d5b-crio-socket\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:13.860038 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.860037    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4h2w\" (UniqueName: \"kubernetes.io/projected/40e67943-b279-4be0-bf71-976537ed7d5b-kube-api-access-d4h2w\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:13.860276 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.860177    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40e67943-b279-4be0-bf71-976537ed7d5b-data-volume\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:13.860276 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.860229    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:13.860378 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.860290    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:13.961439 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.961379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40e67943-b279-4be0-bf71-976537ed7d5b-data-volume\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:13.961439 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.961410    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:13.961439 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.961431    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:13.961823 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:13.961491    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 18:35:13.961823 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:13.961541    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm podName:40e67943-b279-4be0-bf71-976537ed7d5b nodeName:}" failed. No retries permitted until 2026-07-29 18:35:14.461527679 +0000 UTC m=+40.355528058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bqkx8" (UID: "40e67943-b279-4be0-bf71-976537ed7d5b") : configmap "kube-rbac-proxy" not found
Jul 29 18:35:13.961823 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:13.961540    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 18:35:13.961823 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.961595    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40e67943-b279-4be0-bf71-976537ed7d5b-crio-socket\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:13.961823 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:13.961612    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls podName:40e67943-b279-4be0-bf71-976537ed7d5b nodeName:}" failed. No retries permitted until 2026-07-29 18:35:14.461596173 +0000 UTC m=+40.355596561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bqkx8" (UID: "40e67943-b279-4be0-bf71-976537ed7d5b") : secret "insights-runtime-extractor-tls" not found
Jul 29 18:35:13.961823 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.961638    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4h2w\" (UniqueName: \"kubernetes.io/projected/40e67943-b279-4be0-bf71-976537ed7d5b-kube-api-access-d4h2w\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:13.961823 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.961665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40e67943-b279-4be0-bf71-976537ed7d5b-data-volume\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:13.961823 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.961807    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40e67943-b279-4be0-bf71-976537ed7d5b-crio-socket\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:13.972553 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:13.972527    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4h2w\" (UniqueName: \"kubernetes.io/projected/40e67943-b279-4be0-bf71-976537ed7d5b-kube-api-access-d4h2w\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:14.000686 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.000657    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:14.100953 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.100926    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:14.410928 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.410885    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4"]
Jul 29 18:35:14.424863 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.424823    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4"]
Jul 29 18:35:14.425035 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.424960    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4"
Jul 29 18:35:14.427626 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.427597    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 29 18:35:14.427759 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.427675    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 29 18:35:14.427759 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.427749    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 29 18:35:14.427868 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.427832    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-c54dg\""
Jul 29 18:35:14.427960 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.427939    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 29 18:35:14.464303 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.464244    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:14.464303 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.464279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:14.464487 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:14.464388    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 18:35:14.464487 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:14.464403    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 18:35:14.464487 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:14.464465    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm podName:40e67943-b279-4be0-bf71-976537ed7d5b nodeName:}" failed. No retries permitted until 2026-07-29 18:35:15.464447895 +0000 UTC m=+41.358448276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bqkx8" (UID: "40e67943-b279-4be0-bf71-976537ed7d5b") : configmap "kube-rbac-proxy" not found
Jul 29 18:35:14.464487 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:14.464479    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls podName:40e67943-b279-4be0-bf71-976537ed7d5b nodeName:}" failed. No retries permitted until 2026-07-29 18:35:15.464473187 +0000 UTC m=+41.358473567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bqkx8" (UID: "40e67943-b279-4be0-bf71-976537ed7d5b") : secret "insights-runtime-extractor-tls" not found
Jul 29 18:35:14.564631 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.564601    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95020806-e464-4638-9478-c533a95a17f0-config\") pod \"service-ca-operator-5d6c75f84f-x2dq4\" (UID: \"95020806-e464-4638-9478-c533a95a17f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4"
Jul 29 18:35:14.564793 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.564646    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95020806-e464-4638-9478-c533a95a17f0-serving-cert\") pod \"service-ca-operator-5d6c75f84f-x2dq4\" (UID: \"95020806-e464-4638-9478-c533a95a17f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4"
Jul 29 18:35:14.564793 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.564672    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q6df\" (UniqueName: \"kubernetes.io/projected/95020806-e464-4638-9478-c533a95a17f0-kube-api-access-2q6df\") pod \"service-ca-operator-5d6c75f84f-x2dq4\" (UID: \"95020806-e464-4638-9478-c533a95a17f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4"
Jul 29 18:35:14.665405 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.665366    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q6df\" (UniqueName: \"kubernetes.io/projected/95020806-e464-4638-9478-c533a95a17f0-kube-api-access-2q6df\") pod \"service-ca-operator-5d6c75f84f-x2dq4\" (UID: \"95020806-e464-4638-9478-c533a95a17f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4"
Jul 29 18:35:14.665564 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.665500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95020806-e464-4638-9478-c533a95a17f0-config\") pod \"service-ca-operator-5d6c75f84f-x2dq4\" (UID: \"95020806-e464-4638-9478-c533a95a17f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4"
Jul 29 18:35:14.665564 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.665552    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95020806-e464-4638-9478-c533a95a17f0-serving-cert\") pod \"service-ca-operator-5d6c75f84f-x2dq4\" (UID: \"95020806-e464-4638-9478-c533a95a17f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4"
Jul 29 18:35:14.666074 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.666051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95020806-e464-4638-9478-c533a95a17f0-config\") pod \"service-ca-operator-5d6c75f84f-x2dq4\" (UID: \"95020806-e464-4638-9478-c533a95a17f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4"
Jul 29 18:35:14.667838 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.667815    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95020806-e464-4638-9478-c533a95a17f0-serving-cert\") pod \"service-ca-operator-5d6c75f84f-x2dq4\" (UID: \"95020806-e464-4638-9478-c533a95a17f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4"
Jul 29 18:35:14.673178 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.673153    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q6df\" (UniqueName: \"kubernetes.io/projected/95020806-e464-4638-9478-c533a95a17f0-kube-api-access-2q6df\") pod \"service-ca-operator-5d6c75f84f-x2dq4\" (UID: \"95020806-e464-4638-9478-c533a95a17f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4"
Jul 29 18:35:14.732973 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.732900    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-7bqvr"]
Jul 29 18:35:14.736116 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.736097    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4"
Jul 29 18:35:14.759925 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.759902    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-7bqvr"]
Jul 29 18:35:14.760043 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.759988    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7bqvr"
Jul 29 18:35:14.762504 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.762482    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-nv9zt\""
Jul 29 18:35:14.866761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.866727    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqf5k\" (UniqueName: \"kubernetes.io/projected/3f87c81c-72e3-40da-bfd9-fcfa1680bca4-kube-api-access-dqf5k\") pod \"network-check-source-7d88cb55d8-7bqvr\" (UID: \"3f87c81c-72e3-40da-bfd9-fcfa1680bca4\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7bqvr"
Jul 29 18:35:14.882711 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.882675    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4"]
Jul 29 18:35:14.886064 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:35:14.886035    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95020806_e464_4638_9478_c533a95a17f0.slice/crio-138c212b5295de9b97f3dc294a34e5f6043c5dd7577be6cc7e1eba721ef47082 WatchSource:0}: Error finding container 138c212b5295de9b97f3dc294a34e5f6043c5dd7577be6cc7e1eba721ef47082: Status 404 returned error can't find the container with id 138c212b5295de9b97f3dc294a34e5f6043c5dd7577be6cc7e1eba721ef47082
Jul 29 18:35:14.911852 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.911825    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck" event={"ID":"a8e92808-8685-4cd5-b575-e22d16dc483a","Type":"ContainerStarted","Data":"4cc574b8ac8aa35be7086e306b7fcf1705f4a281273e98e77f8231cad658a17c"}
Jul 29 18:35:14.912959 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.912934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4" event={"ID":"95020806-e464-4638-9478-c533a95a17f0","Type":"ContainerStarted","Data":"138c212b5295de9b97f3dc294a34e5f6043c5dd7577be6cc7e1eba721ef47082"}
Jul 29 18:35:14.914385 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.914365    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dw6dv" event={"ID":"5830c3a8-6329-4806-b3f0-1cfb9141886d","Type":"ContainerStarted","Data":"d69a33614f146f6fadc2b308edd1ec4c86f441aaae533f44765ad4efb5cb05b9"}
Jul 29 18:35:14.917590 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.917567    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfw8j" event={"ID":"396aaa29-be70-4f0b-bd94-a3cc455a5cd2","Type":"ContainerStarted","Data":"5b551e49fb0fbf59b87fb2652f7766c8a7800cb2a5dd2000d1b4b7b6955bb07b"}
Jul 29 18:35:14.928377 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.928342    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck" podStartSLOduration=1.681344731 podStartE2EDuration="3.928329532s" totalImagesPullingTime="2.246984801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:14.927424818 +0000 UTC m=+40.821425242" watchObservedRunningTime="2026-07-29 18:35:14.928329532 +0000 UTC m=+40.822329933"
Jul 29 18:35:14.943214 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.943153    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dw6dv" podStartSLOduration=2.886681036 podStartE2EDuration="5.943133078s" totalImagesPullingTime="3.056452042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:14.941492823 +0000 UTC m=+40.835493224" watchObservedRunningTime="2026-07-29 18:35:14.943133078 +0000 UTC m=+40.837133481"
Jul 29 18:35:14.963259 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.963214    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hfw8j" podStartSLOduration=10.063030194 podStartE2EDuration="40.963186392s" totalImagesPullingTime="30.900156198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:14.962792392 +0000 UTC m=+40.856792818" watchObservedRunningTime="2026-07-29 18:35:14.963186392 +0000 UTC m=+40.857186792"
Jul 29 18:35:14.967305 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.967276    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqf5k\" (UniqueName: \"kubernetes.io/projected/3f87c81c-72e3-40da-bfd9-fcfa1680bca4-kube-api-access-dqf5k\") pod \"network-check-source-7d88cb55d8-7bqvr\" (UID: \"3f87c81c-72e3-40da-bfd9-fcfa1680bca4\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7bqvr"
Jul 29 18:35:14.976296 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:14.976268    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqf5k\" (UniqueName: \"kubernetes.io/projected/3f87c81c-72e3-40da-bfd9-fcfa1680bca4-kube-api-access-dqf5k\") pod \"network-check-source-7d88cb55d8-7bqvr\" (UID: \"3f87c81c-72e3-40da-bfd9-fcfa1680bca4\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7bqvr"
Jul 29 18:35:15.068485 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.068454    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7bqvr"
Jul 29 18:35:15.083034 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.083004    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:15.084174 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.084158    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2qlfn_c98ed70e-1aab-43f0-b57d-192dac28efd6/dns-node-resolver/0.log"
Jul 29 18:35:15.140578 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.140543    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:15.173754 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.172892    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-678694966c-d42x9"]
Jul 29 18:35:15.198504 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.198472    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-678694966c-d42x9"]
Jul 29 18:35:15.198632 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.198591    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.201683 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.201660    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 18:35:15.201983 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.201959    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 18:35:15.202102 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.201966    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 18:35:15.202102 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.202024    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v2s6m\""
Jul 29 18:35:15.205945 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.205924    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 18:35:15.223633 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.223611    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-7bqvr"]
Jul 29 18:35:15.226958 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:35:15.226937    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f87c81c_72e3_40da_bfd9_fcfa1680bca4.slice/crio-837564b09e7081a3d97fb9628a6cd899199ba09dcc27ecaeb458c4881095ff38 WatchSource:0}: Error finding container 837564b09e7081a3d97fb9628a6cd899199ba09dcc27ecaeb458c4881095ff38: Status 404 returned error can't find the container with id 837564b09e7081a3d97fb9628a6cd899199ba09dcc27ecaeb458c4881095ff38
Jul 29 18:35:15.371986 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.371913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-ca-trust-extracted\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.371986 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.371946    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-bound-sa-token\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.371986 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.371983    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-certificates\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.372182 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.372031    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-image-registry-private-configuration\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.372182 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.372065    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-trusted-ca\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.372182 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.372088    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8n57\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-kube-api-access-p8n57\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.372182 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.372108    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.372182 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.372125    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-installation-pull-secrets\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.472801 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.472761    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-ca-trust-extracted\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.472967 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.472810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-bound-sa-token\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.472967 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.472856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:15.472967 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.472886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-certificates\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.472967 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.472916    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:15.472967 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.472961    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-image-registry-private-configuration\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.473245 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.473008    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-trusted-ca\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.473245 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.473032    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8n57\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-kube-api-access-p8n57\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.473245 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.473057    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.473245 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.473082    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-installation-pull-secrets\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.473245 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.473190    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-ca-trust-extracted\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.473245 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:15.473008    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 18:35:15.473545 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:15.473316    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls podName:40e67943-b279-4be0-bf71-976537ed7d5b nodeName:}" failed. No retries permitted until 2026-07-29 18:35:17.473290255 +0000 UTC m=+43.367290654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bqkx8" (UID: "40e67943-b279-4be0-bf71-976537ed7d5b") : secret "insights-runtime-extractor-tls" not found
Jul 29 18:35:15.473545 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:15.473055    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 18:35:15.473652 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:15.473554    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm podName:40e67943-b279-4be0-bf71-976537ed7d5b nodeName:}" failed. No retries permitted until 2026-07-29 18:35:17.473532271 +0000 UTC m=+43.367532653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bqkx8" (UID: "40e67943-b279-4be0-bf71-976537ed7d5b") : configmap "kube-rbac-proxy" not found
Jul 29 18:35:15.474054 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.473834    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-certificates\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.474054 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:15.473885    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 18:35:15.474054 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:15.473900    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678694966c-d42x9: secret "image-registry-tls" not found
Jul 29 18:35:15.474054 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:15.473950    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls podName:5dc56110-d7df-4d92-be67-fb6ee4f4ca30 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:15.973933629 +0000 UTC m=+41.867934010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls") pod "image-registry-678694966c-d42x9" (UID: "5dc56110-d7df-4d92-be67-fb6ee4f4ca30") : secret "image-registry-tls" not found
Jul 29 18:35:15.474446 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.474424    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-trusted-ca\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.475873 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.475849    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-image-registry-private-configuration\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.475941 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.475907    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-installation-pull-secrets\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.481502 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.481482    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8n57\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-kube-api-access-p8n57\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.481604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.481541    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-bound-sa-token\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.886347 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.886317    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2qm5s_990500cb-5912-4252-b507-f499bae20618/node-ca/0.log"
Jul 29 18:35:15.923079 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.923043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7bqvr" event={"ID":"3f87c81c-72e3-40da-bfd9-fcfa1680bca4","Type":"ContainerStarted","Data":"837564b09e7081a3d97fb9628a6cd899199ba09dcc27ecaeb458c4881095ff38"}
Jul 29 18:35:15.977891 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:15.977851    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:15.978361 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:15.977997    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 18:35:15.978361 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:15.978012    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678694966c-d42x9: secret "image-registry-tls" not found
Jul 29 18:35:15.978361 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:15.978055    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls podName:5dc56110-d7df-4d92-be67-fb6ee4f4ca30 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:16.978041943 +0000 UTC m=+42.872042322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls") pod "image-registry-678694966c-d42x9" (UID: "5dc56110-d7df-4d92-be67-fb6ee4f4ca30") : secret "image-registry-tls" not found
Jul 29 18:35:16.886799 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:16.886765    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dw6dv_5830c3a8-6329-4806-b3f0-1cfb9141886d/serve-healthcheck-canary/0.log"
Jul 29 18:35:16.986611 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:16.986573    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:16.987144 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:16.986751    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 18:35:16.987144 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:16.986771    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678694966c-d42x9: secret "image-registry-tls" not found
Jul 29 18:35:16.987144 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:16.986846    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls podName:5dc56110-d7df-4d92-be67-fb6ee4f4ca30 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:18.9868227 +0000 UTC m=+44.880823081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls") pod "image-registry-678694966c-d42x9" (UID: "5dc56110-d7df-4d92-be67-fb6ee4f4ca30") : secret "image-registry-tls" not found
Jul 29 18:35:17.169600 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:17.169504    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:17.269910 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:17.269880    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:17.290009 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:17.289979    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:17.290143 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:17.290127    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 18:35:17.290228 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:17.290217    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls podName:e3218c46-a240-4e04-b5a6-1aa3b20524ce nodeName:}" failed. No retries permitted until 2026-07-29 18:35:25.290185074 +0000 UTC m=+51.184185454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls") pod "dns-default-w6dzw" (UID: "e3218c46-a240-4e04-b5a6-1aa3b20524ce") : secret "dns-default-metrics-tls" not found
Jul 29 18:35:17.491534 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:17.491436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:17.491534 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:17.491494    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:17.491750 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:17.491609    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 18:35:17.491750 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:17.491641    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 18:35:17.491750 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:17.491705    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls podName:40e67943-b279-4be0-bf71-976537ed7d5b nodeName:}" failed. No retries permitted until 2026-07-29 18:35:21.491664361 +0000 UTC m=+47.385664740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bqkx8" (UID: "40e67943-b279-4be0-bf71-976537ed7d5b") : secret "insights-runtime-extractor-tls" not found
Jul 29 18:35:17.491750 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:17.491727    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm podName:40e67943-b279-4be0-bf71-976537ed7d5b nodeName:}" failed. No retries permitted until 2026-07-29 18:35:21.491717089 +0000 UTC m=+47.385717471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bqkx8" (UID: "40e67943-b279-4be0-bf71-976537ed7d5b") : configmap "kube-rbac-proxy" not found
Jul 29 18:35:17.632467 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:17.632436    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:17.708071 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:17.708037    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:18.170831 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.170797    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-spkcc"]
Jul 29 18:35:18.201349 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.201317    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-spkcc"]
Jul 29 18:35:18.201490 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.201455    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spkcc"
Jul 29 18:35:18.205099 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.205068    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:18.205673 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.205650    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 29 18:35:18.206148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.206127    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2x682\""
Jul 29 18:35:18.206266 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.206176    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 29 18:35:18.296652 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.296612    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x87hp\" (UniqueName: \"kubernetes.io/projected/42b54cb6-14da-49df-989e-b11119cea71d-kube-api-access-x87hp\") pod \"migrator-6865c9967-spkcc\" (UID: \"42b54cb6-14da-49df-989e-b11119cea71d\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spkcc"
Jul 29 18:35:18.384105 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.384078    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:18.397308 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.397285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x87hp\" (UniqueName: \"kubernetes.io/projected/42b54cb6-14da-49df-989e-b11119cea71d-kube-api-access-x87hp\") pod \"migrator-6865c9967-spkcc\" (UID: \"42b54cb6-14da-49df-989e-b11119cea71d\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spkcc"
Jul 29 18:35:18.405149 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.405127    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x87hp\" (UniqueName: \"kubernetes.io/projected/42b54cb6-14da-49df-989e-b11119cea71d-kube-api-access-x87hp\") pod \"migrator-6865c9967-spkcc\" (UID: \"42b54cb6-14da-49df-989e-b11119cea71d\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spkcc"
Jul 29 18:35:18.512067 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.512044    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spkcc"
Jul 29 18:35:18.665669 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.665636    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-spkcc"]
Jul 29 18:35:18.806287 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:35:18.806185    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b54cb6_14da_49df_989e_b11119cea71d.slice/crio-7a9928082c961d8c079006afafd02bf1ebc7f90ecd73cd9d8ae44f9f88f26c57 WatchSource:0}: Error finding container 7a9928082c961d8c079006afafd02bf1ebc7f90ecd73cd9d8ae44f9f88f26c57: Status 404 returned error can't find the container with id 7a9928082c961d8c079006afafd02bf1ebc7f90ecd73cd9d8ae44f9f88f26c57
Jul 29 18:35:18.931824 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.931792    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4" event={"ID":"95020806-e464-4638-9478-c533a95a17f0","Type":"ContainerStarted","Data":"c137c1d73a0f7bf249561ead84459e484bd6c91352c50971aa2da86bdc555b4a"}
Jul 29 18:35:18.932944 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.932919    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spkcc" event={"ID":"42b54cb6-14da-49df-989e-b11119cea71d","Type":"ContainerStarted","Data":"7a9928082c961d8c079006afafd02bf1ebc7f90ecd73cd9d8ae44f9f88f26c57"}
Jul 29 18:35:18.948266 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:18.948123    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4" podStartSLOduration=1.38035349 podStartE2EDuration="4.948105591s" totalImagesPullingTime="3.567752101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:18.947941312 +0000 UTC m=+44.841941716" watchObservedRunningTime="2026-07-29 18:35:18.948105591 +0000 UTC m=+44.842105994"
Jul 29 18:35:19.003682 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:19.003648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:19.003849 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:19.003827    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 18:35:19.003895 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:19.003853    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678694966c-d42x9: secret "image-registry-tls" not found
Jul 29 18:35:19.003945 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:19.003935    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls podName:5dc56110-d7df-4d92-be67-fb6ee4f4ca30 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:23.003912853 +0000 UTC m=+48.897913234 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls") pod "image-registry-678694966c-d42x9" (UID: "5dc56110-d7df-4d92-be67-fb6ee4f4ca30") : secret "image-registry-tls" not found
Jul 29 18:35:19.912236 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:19.912187    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:19.936329 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:19.936298    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7bqvr" event={"ID":"3f87c81c-72e3-40da-bfd9-fcfa1680bca4","Type":"ContainerStarted","Data":"ed979623c0f52a0cef36ebc262f1a97c423a7a2e82a7801b27552fbba603591b"}
Jul 29 18:35:19.954330 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:19.954273    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7bqvr" podStartSLOduration=2.338433158 podStartE2EDuration="5.954258141s" totalImagesPullingTime="3.615824983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:19.953759965 +0000 UTC m=+45.847760366" watchObservedRunningTime="2026-07-29 18:35:19.954258141 +0000 UTC m=+45.848258553"
Jul 29 18:35:20.013051 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:20.013017    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:20.335794 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:20.335759    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:20.437814 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:20.437779    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:20.621780 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:20.621715    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:20.719007 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:20.718947    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:21.523568 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:21.523538    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:21.524035 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:21.523669    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 18:35:21.524035 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:21.523693    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:21.524035 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:21.523751    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm podName:40e67943-b279-4be0-bf71-976537ed7d5b nodeName:}" failed. No retries permitted until 2026-07-29 18:35:29.523729942 +0000 UTC m=+55.417730343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bqkx8" (UID: "40e67943-b279-4be0-bf71-976537ed7d5b") : configmap "kube-rbac-proxy" not found
Jul 29 18:35:21.524035 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:21.523803    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 18:35:21.524035 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:21.523851    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls podName:40e67943-b279-4be0-bf71-976537ed7d5b nodeName:}" failed. No retries permitted until 2026-07-29 18:35:29.523836953 +0000 UTC m=+55.417837349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bqkx8" (UID: "40e67943-b279-4be0-bf71-976537ed7d5b") : secret "insights-runtime-extractor-tls" not found
Jul 29 18:35:21.731080 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:21.731045    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:21.795717 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:21.795638    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:21.943154 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:21.943114    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spkcc" event={"ID":"42b54cb6-14da-49df-989e-b11119cea71d","Type":"ContainerStarted","Data":"34f48847b9da9090aef6daeb51371ed10d6450418f1ea780bafeddd1f56afee7"}
Jul 29 18:35:21.943340 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:21.943181    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spkcc" event={"ID":"42b54cb6-14da-49df-989e-b11119cea71d","Type":"ContainerStarted","Data":"7bc97791c98ea3d1ac738d6523f7d39bb18720a74e5aa41ad07ed22b21f9beba"}
Jul 29 18:35:21.958605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:21.958559    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spkcc" podStartSLOduration=1.7981421709999998 podStartE2EDuration="3.95854485s" totalImagesPullingTime="2.160402679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:21.957700124 +0000 UTC m=+47.851700525" watchObservedRunningTime="2026-07-29 18:35:21.95854485 +0000 UTC m=+47.852545250"
Jul 29 18:35:22.037588 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:22.037554    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:22.141483 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:22.141455    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:22.245042 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:22.245010    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:22.307077 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:22.307039    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:23.035173 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:23.035136    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:23.035678 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:23.035308    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 18:35:23.035678 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:23.035329    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678694966c-d42x9: secret "image-registry-tls" not found
Jul 29 18:35:23.035678 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:23.035401    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls podName:5dc56110-d7df-4d92-be67-fb6ee4f4ca30 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:31.035380325 +0000 UTC m=+56.929380721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls") pod "image-registry-678694966c-d42x9" (UID: "5dc56110-d7df-4d92-be67-fb6ee4f4ca30") : secret "image-registry-tls" not found
Jul 29 18:35:23.173334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:23.173304    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:23.928419 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:23.928380    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:24.029591 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:24.029557    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:24.471510 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:24.471456    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:25.353583 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:25.353549    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:25.353760 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:25.353658    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 18:35:25.353760 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:25.353711    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls podName:e3218c46-a240-4e04-b5a6-1aa3b20524ce nodeName:}" failed. No retries permitted until 2026-07-29 18:35:41.353696417 +0000 UTC m=+67.247696796 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls") pod "dns-default-w6dzw" (UID: "e3218c46-a240-4e04-b5a6-1aa3b20524ce") : secret "dns-default-metrics-tls" not found
Jul 29 18:35:26.785144 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:26.785101    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:26.945228 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:26.945179    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:27.045457 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:27.045375    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:27.176672 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:27.176641    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:27.274952 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:27.274922    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:27.390839 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:27.390801    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:27.492397 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:27.492367    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:27.662350 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:27.662254    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:28.844124 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:28.844084    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:29.041888 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:29.041842    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:29.247522 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:29.247439    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:29.446454 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:29.446426    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:29.585315 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:29.585285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:29.585492 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:29.585360    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:29.585492 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:29.585459    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 18:35:29.585613 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:35:29.585512    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls podName:40e67943-b279-4be0-bf71-976537ed7d5b nodeName:}" failed. No retries permitted until 2026-07-29 18:35:45.585497313 +0000 UTC m=+71.479497692 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bqkx8" (UID: "40e67943-b279-4be0-bf71-976537ed7d5b") : secret "insights-runtime-extractor-tls" not found
Jul 29 18:35:29.585921 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:29.585894    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40e67943-b279-4be0-bf71-976537ed7d5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:29.648280 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:29.648249    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:30.542215 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:30.542171    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:30.643565 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:30.643530    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:30.744327 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:30.744291    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:31.039046 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:31.039019    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:31.098407 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:31.098379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:31.100678 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:31.100655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls\") pod \"image-registry-678694966c-d42x9\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") " pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:31.109590 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:31.109565    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:31.232774 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:31.232744    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-678694966c-d42x9"]
Jul 29 18:35:31.235634 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:35:31.235592    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc56110_d7df_4d92_be67_fb6ee4f4ca30.slice/crio-ac068904c606ddb03e49a319c3084a2538f81238afed9c67cebed7cc610ccd11 WatchSource:0}: Error finding container ac068904c606ddb03e49a319c3084a2538f81238afed9c67cebed7cc610ccd11: Status 404 returned error can't find the container with id ac068904c606ddb03e49a319c3084a2538f81238afed9c67cebed7cc610ccd11
Jul 29 18:35:31.240762 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:31.240720    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:31.439318 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:31.439290    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:31.499409 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:31.499379    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:31.653279 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:31.652112    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:31.977752 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:31.977721    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678694966c-d42x9" event={"ID":"5dc56110-d7df-4d92-be67-fb6ee4f4ca30","Type":"ContainerStarted","Data":"9fe7edb8fefb427d8464995ada6ad4396071be28bbdbe680122b927ddbe0e16a"}
Jul 29 18:35:31.977931 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:31.977769    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678694966c-d42x9" event={"ID":"5dc56110-d7df-4d92-be67-fb6ee4f4ca30","Type":"ContainerStarted","Data":"ac068904c606ddb03e49a319c3084a2538f81238afed9c67cebed7cc610ccd11"}
Jul 29 18:35:31.977931 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:31.977836    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:31.996181 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:31.996127    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-678694966c-d42x9" podStartSLOduration=16.996110198 podStartE2EDuration="16.996110198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:31.995747912 +0000 UTC m=+57.889748339" watchObservedRunningTime="2026-07-29 18:35:31.996110198 +0000 UTC m=+57.890110599"
Jul 29 18:35:32.610983 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:32.610952    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:32.711765 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:32.711732    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:32.850154 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:32.850125    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:33.043183 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:33.043103    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:33.143820 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:33.143792    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:33.244166 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:33.244135    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:33.305522 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:33.305451    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:33.441412 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:33.441384    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:33.541551 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:33.541523    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:33.642340 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:33.642310    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:33.742638 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:33.742602    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:34.126912 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:34.126878    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:34.209851 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:34.209818    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:34.289690 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:34.289662    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:34.387787 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:34.387713    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:34.522992 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:34.522959    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:34.624104 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:34.624068    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:34.688215 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:34.688133    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:34.839917 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:34.839883    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:34.889586 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:34.889556    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlsrd"
Jul 29 18:35:34.944414 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:34.944346    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:35.063888 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:35.063858    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:35.168169 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:35.168142    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:35.266847 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:35.266768    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:35.419372 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:35.419330    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:35.941778 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:35.941746    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:36.043020 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:36.042988    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:36.248082 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:36.247995    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:36.647246 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:36.647214    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:36.844403 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:36.844372    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:37.056911 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:37.056881    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:37.440031 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:37.439948    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:37.848472 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:37.848436    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:38.037769 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:38.037733    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:40.267547 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.267501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:35:40.270310 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.270293    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 18:35:40.279821 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.279787    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/244afbce-35b6-4432-a874-d7081b20c82e-original-pull-secret\") pod \"global-pull-secret-syncer-rrsh7\" (UID: \"244afbce-35b6-4432-a874-d7081b20c82e\") " pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:35:40.293856 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.293830    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrsh7"
Jul 29 18:35:40.368535 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.368505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs\") pod \"network-metrics-daemon-2jvrj\" (UID: \"7154111a-1f26-4f41-8ba6-94617dcca1b3\") " pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:35:40.368639 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.368562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdq9w\" (UniqueName: \"kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w\") pod \"network-check-target-44mt4\" (UID: \"7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e\") " pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:35:40.371452 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.371424    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdq9w\" (UniqueName: \"kubernetes.io/projected/7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e-kube-api-access-bdq9w\") pod \"network-check-target-44mt4\" (UID: \"7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e\") " pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:35:40.371590 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.371486    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 18:35:40.380834 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.380813    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7154111a-1f26-4f41-8ba6-94617dcca1b3-metrics-certs\") pod \"network-metrics-daemon-2jvrj\" (UID: \"7154111a-1f26-4f41-8ba6-94617dcca1b3\") " pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:35:40.395182 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.395161    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b24qk\""
Jul 29 18:35:40.403323 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.403302    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:35:40.412388 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.412366    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rrsh7"]
Jul 29 18:35:40.415213 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:35:40.415179    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244afbce_35b6_4432_a874_d7081b20c82e.slice/crio-cd10447304a7970ffc750f49262cb7ed69a3ece5f3b23dcece8fd73231f0b94b WatchSource:0}: Error finding container cd10447304a7970ffc750f49262cb7ed69a3ece5f3b23dcece8fd73231f0b94b: Status 404 returned error can't find the container with id cd10447304a7970ffc750f49262cb7ed69a3ece5f3b23dcece8fd73231f0b94b
Jul 29 18:35:40.523097 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.523022    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-44mt4"]
Jul 29 18:35:40.525672 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:35:40.525642    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7343dc_8e04_45c7_8fd4_ffd0bcb3c96e.slice/crio-1cbf8f3004fbe6c0b6a1303ec58003c4cf26e8d4a1564709454d487fe3d44403 WatchSource:0}: Error finding container 1cbf8f3004fbe6c0b6a1303ec58003c4cf26e8d4a1564709454d487fe3d44403: Status 404 returned error can't find the container with id 1cbf8f3004fbe6c0b6a1303ec58003c4cf26e8d4a1564709454d487fe3d44403
Jul 29 18:35:40.600906 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.600885    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g7g2r\""
Jul 29 18:35:40.609099 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.609080    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jvrj"
Jul 29 18:35:40.731069 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:40.731029    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2jvrj"]
Jul 29 18:35:40.733791 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:35:40.733756    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7154111a_1f26_4f41_8ba6_94617dcca1b3.slice/crio-39d94650218cb6f53800f5b332a2e9dd4ac66b76d8bc163417542f304bd48019 WatchSource:0}: Error finding container 39d94650218cb6f53800f5b332a2e9dd4ac66b76d8bc163417542f304bd48019: Status 404 returned error can't find the container with id 39d94650218cb6f53800f5b332a2e9dd4ac66b76d8bc163417542f304bd48019
Jul 29 18:35:41.010796 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:41.010765    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2jvrj" event={"ID":"7154111a-1f26-4f41-8ba6-94617dcca1b3","Type":"ContainerStarted","Data":"39d94650218cb6f53800f5b332a2e9dd4ac66b76d8bc163417542f304bd48019"}
Jul 29 18:35:41.012446 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:41.012416    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-44mt4" event={"ID":"7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e","Type":"ContainerStarted","Data":"6ee47a5e0f3707b210b3034ce4ac0240ac8cbb4c5d50d46a7cb7a63d1aeb1afa"}
Jul 29 18:35:41.012587 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:41.012463    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-44mt4" event={"ID":"7f7343dc-8e04-45c7-8fd4-ffd0bcb3c96e","Type":"ContainerStarted","Data":"1cbf8f3004fbe6c0b6a1303ec58003c4cf26e8d4a1564709454d487fe3d44403"}
Jul 29 18:35:41.012587 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:41.012491    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:35:41.013677 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:41.013645    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rrsh7" event={"ID":"244afbce-35b6-4432-a874-d7081b20c82e","Type":"ContainerStarted","Data":"cd10447304a7970ffc750f49262cb7ed69a3ece5f3b23dcece8fd73231f0b94b"}
Jul 29 18:35:41.029478 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:41.029430    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-44mt4" podStartSLOduration=67.029416169 podStartE2EDuration="1m7.029416169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:41.02841397 +0000 UTC m=+66.922414402" watchObservedRunningTime="2026-07-29 18:35:41.029416169 +0000 UTC m=+66.923416569"
Jul 29 18:35:41.185312 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:41.185281    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:41.269323 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:41.269240    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:41.378433 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:41.378375    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:41.381055 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:41.381027    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3218c46-a240-4e04-b5a6-1aa3b20524ce-metrics-tls\") pod \"dns-default-w6dzw\" (UID: \"e3218c46-a240-4e04-b5a6-1aa3b20524ce\") " pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:41.638745 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:41.638705    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hgmvm\""
Jul 29 18:35:41.646741 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:41.646701    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:41.803404 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:41.803374    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w6dzw"]
Jul 29 18:35:41.807009 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:35:41.806986    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3218c46_a240_4e04_b5a6_1aa3b20524ce.slice/crio-cb78de3094399cd233eb3a912f9bb564201db8c9be0e72c95b0cdfc2f80fd36f WatchSource:0}: Error finding container cb78de3094399cd233eb3a912f9bb564201db8c9be0e72c95b0cdfc2f80fd36f: Status 404 returned error can't find the container with id cb78de3094399cd233eb3a912f9bb564201db8c9be0e72c95b0cdfc2f80fd36f
Jul 29 18:35:42.019976 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:42.019922    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2jvrj" event={"ID":"7154111a-1f26-4f41-8ba6-94617dcca1b3","Type":"ContainerStarted","Data":"bf2740afa56029d0b7fdfd725a8d3e58dd7cbeb2df92acf55316f0e394af2748"}
Jul 29 18:35:42.020133 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:42.020034    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2jvrj" event={"ID":"7154111a-1f26-4f41-8ba6-94617dcca1b3","Type":"ContainerStarted","Data":"8573ea820c6e9f164257ed44d5ff7c7ef0a32ee9a3dd5891744440c34388088d"}
Jul 29 18:35:42.021264 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:42.021232    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w6dzw" event={"ID":"e3218c46-a240-4e04-b5a6-1aa3b20524ce","Type":"ContainerStarted","Data":"cb78de3094399cd233eb3a912f9bb564201db8c9be0e72c95b0cdfc2f80fd36f"}
Jul 29 18:35:42.039016 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:42.038961    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2jvrj" podStartSLOduration=67.069972563 podStartE2EDuration="1m8.038942036s" totalImagesPullingTime="968.969473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:42.037528939 +0000 UTC m=+67.931529361" watchObservedRunningTime="2026-07-29 18:35:42.038942036 +0000 UTC m=+67.932942439"
Jul 29 18:35:44.315911 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.315878    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"]
Jul 29 18:35:44.335492 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.335469    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.338153 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.338104    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"]
Jul 29 18:35:44.338630 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.338569    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 29 18:35:44.338630 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.338572    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 29 18:35:44.338926 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.338910    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 29 18:35:44.339370 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.339152    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 18:35:44.339370 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.339165    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 18:35:44.339370 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.339241    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 18:35:44.339370 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.339257    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 29 18:35:44.339567 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.339502    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-678694966c-d42x9"]
Jul 29 18:35:44.404758 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.404723    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b41ae1f5-1144-409b-a6a6-5f5b77475d67-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.404912 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.404763    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b41ae1f5-1144-409b-a6a6-5f5b77475d67-hub\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.404912 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.404780    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41ae1f5-1144-409b-a6a6-5f5b77475d67-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.404912 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.404802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b41ae1f5-1144-409b-a6a6-5f5b77475d67-ca\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.404912 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.404901    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b41ae1f5-1144-409b-a6a6-5f5b77475d67-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.405072 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.404929    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrn2c\" (UniqueName: \"kubernetes.io/projected/b41ae1f5-1144-409b-a6a6-5f5b77475d67-kube-api-access-lrn2c\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.482254 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.482223    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:44.506260 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.506228    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b41ae1f5-1144-409b-a6a6-5f5b77475d67-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.506419 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.506279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b41ae1f5-1144-409b-a6a6-5f5b77475d67-hub\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.506419 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.506398    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41ae1f5-1144-409b-a6a6-5f5b77475d67-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.506533 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.506443    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b41ae1f5-1144-409b-a6a6-5f5b77475d67-ca\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.506533 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.506522    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b41ae1f5-1144-409b-a6a6-5f5b77475d67-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.506634 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.506568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrn2c\" (UniqueName: \"kubernetes.io/projected/b41ae1f5-1144-409b-a6a6-5f5b77475d67-kube-api-access-lrn2c\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.507223 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.507173    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41ae1f5-1144-409b-a6a6-5f5b77475d67-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.508820 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.508793    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b41ae1f5-1144-409b-a6a6-5f5b77475d67-ca\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.508926 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.508908    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b41ae1f5-1144-409b-a6a6-5f5b77475d67-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.509134 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.509118    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b41ae1f5-1144-409b-a6a6-5f5b77475d67-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.509289 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.509269    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b41ae1f5-1144-409b-a6a6-5f5b77475d67-hub\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.527563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.527541    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrn2c\" (UniqueName: \"kubernetes.io/projected/b41ae1f5-1144-409b-a6a6-5f5b77475d67-kube-api-access-lrn2c\") pod \"cluster-proxy-proxy-agent-6577ccf748-lfb5h\" (UID: \"b41ae1f5-1144-409b-a6a6-5f5b77475d67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.615289 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.615217    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:44.654077 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.654049    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"
Jul 29 18:35:44.719567 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.719528    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:44.818494 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.818469    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:44.825861 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.825803    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h"]
Jul 29 18:35:44.828489 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:35:44.828466    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb41ae1f5_1144_409b_a6a6_5f5b77475d67.slice/crio-597acc96ffc9b2417481fcb7071d95bae60fceab309e5cbd830af2ce0107b32b WatchSource:0}: Error finding container 597acc96ffc9b2417481fcb7071d95bae60fceab309e5cbd830af2ce0107b32b: Status 404 returned error can't find the container with id 597acc96ffc9b2417481fcb7071d95bae60fceab309e5cbd830af2ce0107b32b
Jul 29 18:35:44.921154 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:44.921061    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:45.040452 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:45.040409    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rrsh7" event={"ID":"244afbce-35b6-4432-a874-d7081b20c82e","Type":"ContainerStarted","Data":"dc3b3a65abaf8754437be1e1c7100f77391b4c79888165e80a158f73e6471ddd"}
Jul 29 18:35:45.041566 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:45.041538    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h" event={"ID":"b41ae1f5-1144-409b-a6a6-5f5b77475d67","Type":"ContainerStarted","Data":"597acc96ffc9b2417481fcb7071d95bae60fceab309e5cbd830af2ce0107b32b"}
Jul 29 18:35:45.056093 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:45.056046    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rrsh7" podStartSLOduration=66.790683582 podStartE2EDuration="1m11.056005838s" totalImagesPullingTime="4.265322256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:45.054680398 +0000 UTC m=+70.948680799" watchObservedRunningTime="2026-07-29 18:35:45.056005838 +0000 UTC m=+70.950006240"
Jul 29 18:35:45.388410 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:45.388380    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:45.457464 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:45.457432    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:45.617296 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:45.617259    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:45.619658 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:45.619632    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40e67943-b279-4be0-bf71-976537ed7d5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bqkx8\" (UID: \"40e67943-b279-4be0-bf71-976537ed7d5b\") " pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:45.758763 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:45.758669    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:45.832713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:45.832690    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8px9m\""
Jul 29 18:35:45.840805 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:45.840779    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bqkx8"
Jul 29 18:35:45.858611 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:45.858583    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:45.997021 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:45.996993    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bqkx8"]
Jul 29 18:35:46.001118 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:35:46.001092    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e67943_b279_4be0_bf71_976537ed7d5b.slice/crio-fb409231a02d55293a5afa96ad88242c75caa55f35fe596ff484b004dc5d4ddf WatchSource:0}: Error finding container fb409231a02d55293a5afa96ad88242c75caa55f35fe596ff484b004dc5d4ddf: Status 404 returned error can't find the container with id fb409231a02d55293a5afa96ad88242c75caa55f35fe596ff484b004dc5d4ddf
Jul 29 18:35:46.047031 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:46.046990    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w6dzw" event={"ID":"e3218c46-a240-4e04-b5a6-1aa3b20524ce","Type":"ContainerStarted","Data":"09358b78fc613aba6fa079cb7f4e5bbb34154c805446cf09d1c1075faae528aa"}
Jul 29 18:35:46.047174 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:46.047051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w6dzw" event={"ID":"e3218c46-a240-4e04-b5a6-1aa3b20524ce","Type":"ContainerStarted","Data":"ed08c7a3215cc849d2fd28d1239f82d102b93fa1f8dfd75e2242181fbd6e8e58"}
Jul 29 18:35:46.047174 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:46.047122    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:46.048243 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:46.048221    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bqkx8" event={"ID":"40e67943-b279-4be0-bf71-976537ed7d5b","Type":"ContainerStarted","Data":"fb409231a02d55293a5afa96ad88242c75caa55f35fe596ff484b004dc5d4ddf"}
Jul 29 18:35:46.065187 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:46.065142    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w6dzw" podStartSLOduration=33.074161117 podStartE2EDuration="37.065130604s" totalImagesPullingTime="3.990969487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:46.064153735 +0000 UTC m=+71.958154137" watchObservedRunningTime="2026-07-29 18:35:46.065130604 +0000 UTC m=+71.959131005"
Jul 29 18:35:46.330498 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:46.330461    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:46.777540 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:46.777451    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:47.059685 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:47.059619    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:47.156611 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:47.156576    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:47.266718 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:47.266615    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:47.577914 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:47.577850    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:47.679056 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:47.678963    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:49.144537 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:49.144467    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:49.244674 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:49.244640    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:50.276637 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:50.276608    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:50.344265 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:50.344231    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:50.401736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:50.401700    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:50.457568 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:50.457535    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:51.754929 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:51.754892    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:51.852399 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:51.852362    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:53.076351 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:53.076313    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h" event={"ID":"b41ae1f5-1144-409b-a6a6-5f5b77475d67","Type":"ContainerStarted","Data":"cd7812c8a06692507c5e31f7ebc920d66931603571935c42c3ceb0aba7ff2968"}
Jul 29 18:35:53.076792 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:53.076375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h" event={"ID":"b41ae1f5-1144-409b-a6a6-5f5b77475d67","Type":"ContainerStarted","Data":"9978e5240cc8ae2360ec0f1cd4faf9c3e14888f5d473e66041e006bb9a488a2f"}
Jul 29 18:35:53.076792 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:53.076397    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h" event={"ID":"b41ae1f5-1144-409b-a6a6-5f5b77475d67","Type":"ContainerStarted","Data":"023b755b8a3bf2756e15fc4178388c73d51783cfadd1add22f8bd03732fde5e1"}
Jul 29 18:35:53.078760 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:53.078731    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bqkx8" event={"ID":"40e67943-b279-4be0-bf71-976537ed7d5b","Type":"ContainerStarted","Data":"18bffd6c0d77d5a6bec234144b5112d0333db5032f332dc20d70a1ca34100495"}
Jul 29 18:35:53.095091 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:53.095049    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6577ccf748-lfb5h" podStartSLOduration=1.273899355 podStartE2EDuration="9.095036091s" totalImagesPullingTime="7.821136736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:53.094063298 +0000 UTC m=+78.988063712" watchObservedRunningTime="2026-07-29 18:35:53.095036091 +0000 UTC m=+78.989036492"
Jul 29 18:35:53.865130 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:53.865096    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:53.924851 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:53.924766    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:54.345656 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:54.345624    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:35:54.346495 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:54.346469    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:54.447618 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:54.447582    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:54.839054 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:54.839006    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:56.055945 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.055815    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w6dzw"
Jul 29 18:35:56.480218 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.480148    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:56.568365 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.568328    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:56.676441 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.676408    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:56.733472 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.733393    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9q7nb"]
Jul 29 18:35:56.736829 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.736812    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.739054 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.739033    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 18:35:56.739264 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.739248    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 18:35:56.739387 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.739341    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2fkkq\""
Jul 29 18:35:56.739387 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.739356    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 18:35:56.739387 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.739382    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 18:35:56.740250 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.740229    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 18:35:56.740406 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.740230    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 18:35:56.772545 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.772514    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:56.805947 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.805922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-tls\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.806072 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.805955    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7745c9cd-f92a-44e3-b485-96782dd3de5a-sys\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.806072 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.805975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.806072 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.805992    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7745c9cd-f92a-44e3-b485-96782dd3de5a-root\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.806072 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.806045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-textfile\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.806072 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.806061    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-wtmp\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.806298 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.806076    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7745c9cd-f92a-44e3-b485-96782dd3de5a-metrics-client-ca\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.806298 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.806125    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qljhb\" (UniqueName: \"kubernetes.io/projected/7745c9cd-f92a-44e3-b485-96782dd3de5a-kube-api-access-qljhb\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.806298 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.806182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-accelerators-collector-config\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.874513 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.874481    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:56.906668 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.906642    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-textfile\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.906813 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.906671    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-wtmp\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.906813 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.906691    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7745c9cd-f92a-44e3-b485-96782dd3de5a-metrics-client-ca\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.906813 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.906721    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qljhb\" (UniqueName: \"kubernetes.io/projected/7745c9cd-f92a-44e3-b485-96782dd3de5a-kube-api-access-qljhb\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.906813 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.906774    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-accelerators-collector-config\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.907021 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.906823    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-tls\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.907021 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.906851    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-wtmp\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.907021 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.906850    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7745c9cd-f92a-44e3-b485-96782dd3de5a-sys\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.907021 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.906890    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7745c9cd-f92a-44e3-b485-96782dd3de5a-sys\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.907021 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.906905    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.907021 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.906936    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7745c9cd-f92a-44e3-b485-96782dd3de5a-root\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.907021 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.906998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7745c9cd-f92a-44e3-b485-96782dd3de5a-root\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.907392 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.907019    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-textfile\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.907429 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.907395    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-accelerators-collector-config\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.907474 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.907458    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7745c9cd-f92a-44e3-b485-96782dd3de5a-metrics-client-ca\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.909111 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.909093    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.909230 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.909217    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7745c9cd-f92a-44e3-b485-96782dd3de5a-node-exporter-tls\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:56.917350 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:56.917323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qljhb\" (UniqueName: \"kubernetes.io/projected/7745c9cd-f92a-44e3-b485-96782dd3de5a-kube-api-access-qljhb\") pod \"node-exporter-9q7nb\" (UID: \"7745c9cd-f92a-44e3-b485-96782dd3de5a\") " pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:57.046381 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:57.046304    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9q7nb"
Jul 29 18:35:57.057770 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:35:57.057693    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7745c9cd_f92a_44e3_b485_96782dd3de5a.slice/crio-470e0ec931b4309da280cac6984582aaac2a6381c08f94c29a54fa76c13fa3cb WatchSource:0}: Error finding container 470e0ec931b4309da280cac6984582aaac2a6381c08f94c29a54fa76c13fa3cb: Status 404 returned error can't find the container with id 470e0ec931b4309da280cac6984582aaac2a6381c08f94c29a54fa76c13fa3cb
Jul 29 18:35:57.097750 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:57.097708    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bqkx8" event={"ID":"40e67943-b279-4be0-bf71-976537ed7d5b","Type":"ContainerStarted","Data":"09d61932797ec205211d0a5f5dba5a59ef2a6bc67590a05c981858ee108efd90"}
Jul 29 18:35:57.098993 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:57.098971    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q7nb" event={"ID":"7745c9cd-f92a-44e3-b485-96782dd3de5a","Type":"ContainerStarted","Data":"470e0ec931b4309da280cac6984582aaac2a6381c08f94c29a54fa76c13fa3cb"}
Jul 29 18:35:57.637280 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:57.637241    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:57.737683 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:57.737630    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:57.813018 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:57.812972    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:58.202372 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:58.202333    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:58.773620 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:58.773577    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:58.862403 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:58.862369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:59.108354 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:59.108310    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q7nb" event={"ID":"7745c9cd-f92a-44e3-b485-96782dd3de5a","Type":"ContainerStarted","Data":"2f7bc0d57132c22fe83806f3b5c4bc5283d037a7f07332c83de06c89894d3931"}
Jul 29 18:35:59.709517 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:59.709476    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:59.809865 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:59.809829    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:59.881768 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:59.881736    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:35:59.948996 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:35:59.948963    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:00.048426 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:00.048401    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:00.113587 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:00.113558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bqkx8" event={"ID":"40e67943-b279-4be0-bf71-976537ed7d5b","Type":"ContainerStarted","Data":"9f52abd41098731e355bf0a142257c8777b71b952e947ce61efe0e43b07e3e8e"}
Jul 29 18:36:00.114986 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:00.114963    2560 generic.go:356] "Generic (PLEG): container finished" podID="7745c9cd-f92a-44e3-b485-96782dd3de5a" containerID="2f7bc0d57132c22fe83806f3b5c4bc5283d037a7f07332c83de06c89894d3931" exitCode=0
Jul 29 18:36:00.115078 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:00.115022    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q7nb" event={"ID":"7745c9cd-f92a-44e3-b485-96782dd3de5a","Type":"ContainerDied","Data":"2f7bc0d57132c22fe83806f3b5c4bc5283d037a7f07332c83de06c89894d3931"}
Jul 29 18:36:00.135381 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:00.135310    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bqkx8" podStartSLOduration=33.234756849 podStartE2EDuration="47.135289061s" totalImagesPullingTime="13.900532212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:35:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:36:00.133962184 +0000 UTC m=+86.027962618" watchObservedRunningTime="2026-07-29 18:36:00.135289061 +0000 UTC m=+86.029289465"
Jul 29 18:36:00.140149 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:00.140114    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:01.120068 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:01.120028    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q7nb" event={"ID":"7745c9cd-f92a-44e3-b485-96782dd3de5a","Type":"ContainerStarted","Data":"fec60ac2511ea7bf079d50b34a8c580a1fb3a735103c8264e9cc724dc7b488f6"}
Jul 29 18:36:01.120464 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:01.120122    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q7nb" event={"ID":"7745c9cd-f92a-44e3-b485-96782dd3de5a","Type":"ContainerStarted","Data":"fbd3b4e278816dd1249f68f6bc44e59bdbade4809355d43ea51091fa0bfc157f"}
Jul 29 18:36:01.141020 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:01.140973    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9q7nb" podStartSLOduration=3.8611027829999998 podStartE2EDuration="5.140958141s" totalImagesPullingTime="1.21148179s" totalInitContainerRuntime="68.373568ms" isStatefulPod=true podCreationTimestamp="2026-07-29 18:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:36:01.139991799 +0000 UTC m=+87.033992200" watchObservedRunningTime="2026-07-29 18:36:01.140958141 +0000 UTC m=+87.034958542"
Jul 29 18:36:01.593803 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:01.593768    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:01.650015 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:01.649986    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:02.275708 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:02.275676    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:02.376214 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:02.376165    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:03.013164 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:03.013134    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:03.256279 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:03.256245    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:03.356438 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:03.356395    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:07.832814 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:07.832778    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:07.894213 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:07.894164    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:07.963570 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:07.963537    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:08.065100 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:08.065067    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:08.814308 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:08.814277    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:08.914350 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:08.914315    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:09.358360 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.358319    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-678694966c-d42x9" podUID="5dc56110-d7df-4d92-be67-fb6ee4f4ca30" containerName="registry" containerID="cri-o://9fe7edb8fefb427d8464995ada6ad4396071be28bbdbe680122b927ddbe0e16a" gracePeriod=30
Jul 29 18:36:09.592559 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.592533    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:36:09.703100 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.703012    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls\") pod \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") "
Jul 29 18:36:09.703100 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.703049    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-bound-sa-token\") pod \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") "
Jul 29 18:36:09.703100 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.703069    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8n57\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-kube-api-access-p8n57\") pod \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") "
Jul 29 18:36:09.703100 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.703095    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-trusted-ca\") pod \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") "
Jul 29 18:36:09.703427 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.703120    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-installation-pull-secrets\") pod \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") "
Jul 29 18:36:09.703427 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.703151    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-certificates\") pod \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") "
Jul 29 18:36:09.703427 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.703256    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-ca-trust-extracted\") pod \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") "
Jul 29 18:36:09.703427 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.703293    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-image-registry-private-configuration\") pod \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\" (UID: \"5dc56110-d7df-4d92-be67-fb6ee4f4ca30\") "
Jul 29 18:36:09.704179 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.703627    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5dc56110-d7df-4d92-be67-fb6ee4f4ca30" (UID: "5dc56110-d7df-4d92-be67-fb6ee4f4ca30"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 18:36:09.704179 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.703656    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5dc56110-d7df-4d92-be67-fb6ee4f4ca30" (UID: "5dc56110-d7df-4d92-be67-fb6ee4f4ca30"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 18:36:09.705677 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.705638    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-kube-api-access-p8n57" (OuterVolumeSpecName: "kube-api-access-p8n57") pod "5dc56110-d7df-4d92-be67-fb6ee4f4ca30" (UID: "5dc56110-d7df-4d92-be67-fb6ee4f4ca30"). InnerVolumeSpecName "kube-api-access-p8n57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:36:09.705812 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.705692    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5dc56110-d7df-4d92-be67-fb6ee4f4ca30" (UID: "5dc56110-d7df-4d92-be67-fb6ee4f4ca30"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:36:09.705875 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.705810    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5dc56110-d7df-4d92-be67-fb6ee4f4ca30" (UID: "5dc56110-d7df-4d92-be67-fb6ee4f4ca30"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:36:09.706008 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.705990    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5dc56110-d7df-4d92-be67-fb6ee4f4ca30" (UID: "5dc56110-d7df-4d92-be67-fb6ee4f4ca30"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:36:09.706084 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.706059    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5dc56110-d7df-4d92-be67-fb6ee4f4ca30" (UID: "5dc56110-d7df-4d92-be67-fb6ee4f4ca30"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:36:09.712318 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.712292    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5dc56110-d7df-4d92-be67-fb6ee4f4ca30" (UID: "5dc56110-d7df-4d92-be67-fb6ee4f4ca30"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:36:09.804460 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.804413    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-tls\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:36:09.804460 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.804456    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-bound-sa-token\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:36:09.804460 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.804466    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8n57\" (UniqueName: \"kubernetes.io/projected/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-kube-api-access-p8n57\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:36:09.804460 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.804476    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-trusted-ca\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:36:09.804699 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.804484    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-installation-pull-secrets\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:36:09.804699 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.804492    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-registry-certificates\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:36:09.804699 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.804502    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-ca-trust-extracted\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:36:09.804699 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:09.804512    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5dc56110-d7df-4d92-be67-fb6ee4f4ca30-image-registry-private-configuration\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:36:10.136244 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:10.136188    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:10.156420 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:10.156388    2560 generic.go:356] "Generic (PLEG): container finished" podID="5dc56110-d7df-4d92-be67-fb6ee4f4ca30" containerID="9fe7edb8fefb427d8464995ada6ad4396071be28bbdbe680122b927ddbe0e16a" exitCode=0
Jul 29 18:36:10.156574 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:10.156440    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678694966c-d42x9"
Jul 29 18:36:10.156574 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:10.156469    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678694966c-d42x9" event={"ID":"5dc56110-d7df-4d92-be67-fb6ee4f4ca30","Type":"ContainerDied","Data":"9fe7edb8fefb427d8464995ada6ad4396071be28bbdbe680122b927ddbe0e16a"}
Jul 29 18:36:10.156574 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:10.156535    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678694966c-d42x9" event={"ID":"5dc56110-d7df-4d92-be67-fb6ee4f4ca30","Type":"ContainerDied","Data":"ac068904c606ddb03e49a319c3084a2538f81238afed9c67cebed7cc610ccd11"}
Jul 29 18:36:10.156574 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:10.156555    2560 scope.go:117] "RemoveContainer" containerID="9fe7edb8fefb427d8464995ada6ad4396071be28bbdbe680122b927ddbe0e16a"
Jul 29 18:36:10.164902 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:10.164884    2560 scope.go:117] "RemoveContainer" containerID="9fe7edb8fefb427d8464995ada6ad4396071be28bbdbe680122b927ddbe0e16a"
Jul 29 18:36:10.165160 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:36:10.165141    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fe7edb8fefb427d8464995ada6ad4396071be28bbdbe680122b927ddbe0e16a\": container with ID starting with 9fe7edb8fefb427d8464995ada6ad4396071be28bbdbe680122b927ddbe0e16a not found: ID does not exist" containerID="9fe7edb8fefb427d8464995ada6ad4396071be28bbdbe680122b927ddbe0e16a"
Jul 29 18:36:10.165247 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:10.165167    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fe7edb8fefb427d8464995ada6ad4396071be28bbdbe680122b927ddbe0e16a"} err="failed to get container status \"9fe7edb8fefb427d8464995ada6ad4396071be28bbdbe680122b927ddbe0e16a\": rpc error: code = NotFound desc = could not find container \"9fe7edb8fefb427d8464995ada6ad4396071be28bbdbe680122b927ddbe0e16a\": container with ID starting with 9fe7edb8fefb427d8464995ada6ad4396071be28bbdbe680122b927ddbe0e16a not found: ID does not exist"
Jul 29 18:36:10.178443 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:10.178419    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-678694966c-d42x9"]
Jul 29 18:36:10.182036 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:10.182015    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-678694966c-d42x9"]
Jul 29 18:36:10.239439 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:10.239410    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:10.686715 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:10.686677    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dc56110-d7df-4d92-be67-fb6ee4f4ca30" path="/var/lib/kubelet/pods/5dc56110-d7df-4d92-be67-fb6ee4f4ca30/volumes"
Jul 29 18:36:12.024320 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:12.024287    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-44mt4"
Jul 29 18:36:12.133729 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:12.133696    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:12.238267 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:12.238237    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:17.616666 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:17.616629    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:18.086433 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:18.086402    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:19.217118 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:19.217080    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:19.330354 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:19.330324    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:23.400277 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:23.400240    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:23.455090 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:23.455059    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:23.510858 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:23.510823    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:25.208830 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:25.208796    2560 generic.go:356] "Generic (PLEG): container finished" podID="95020806-e464-4638-9478-c533a95a17f0" containerID="c137c1d73a0f7bf249561ead84459e484bd6c91352c50971aa2da86bdc555b4a" exitCode=0
Jul 29 18:36:25.209224 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:25.208840    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4" event={"ID":"95020806-e464-4638-9478-c533a95a17f0","Type":"ContainerDied","Data":"c137c1d73a0f7bf249561ead84459e484bd6c91352c50971aa2da86bdc555b4a"}
Jul 29 18:36:25.209224 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:25.209157    2560 scope.go:117] "RemoveContainer" containerID="c137c1d73a0f7bf249561ead84459e484bd6c91352c50971aa2da86bdc555b4a"
Jul 29 18:36:25.230629 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:25.230602    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:25.293920 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:25.293894    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:25.395322 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:25.395289    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:26.213964 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:26.213931    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x2dq4" event={"ID":"95020806-e464-4638-9478-c533a95a17f0","Type":"ContainerStarted","Data":"bc2c8595d032992ce084ee119be40fcb55c18de2c5b6ab6c206c118fdffaeb1d"}
Jul 29 18:36:32.613629 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:32.613590    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:32.673653 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:32.673623    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:36:37.251015 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:37.250977    2560 generic.go:356] "Generic (PLEG): container finished" podID="a8e92808-8685-4cd5-b575-e22d16dc483a" containerID="4cc574b8ac8aa35be7086e306b7fcf1705f4a281273e98e77f8231cad658a17c" exitCode=0
Jul 29 18:36:37.251424 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:37.251026    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck" event={"ID":"a8e92808-8685-4cd5-b575-e22d16dc483a","Type":"ContainerDied","Data":"4cc574b8ac8aa35be7086e306b7fcf1705f4a281273e98e77f8231cad658a17c"}
Jul 29 18:36:37.251424 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:37.251332    2560 scope.go:117] "RemoveContainer" containerID="4cc574b8ac8aa35be7086e306b7fcf1705f4a281273e98e77f8231cad658a17c"
Jul 29 18:36:38.256120 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:36:38.256082    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck" event={"ID":"a8e92808-8685-4cd5-b575-e22d16dc483a","Type":"ContainerStarted","Data":"76b1a110bbcb0d172a8dd8a029425b13f1f31ec46bad2968ca488109c903cbd2"}
Jul 29 18:37:15.998293 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:37:15.998252    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:37:16.112068 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:37:16.112039    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:37:25.479746 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:37:25.479709    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:37:36.728311 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:37:36.728237    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:37:36.781585 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:37:36.781549    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:37:36.840056 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:37:36.840024    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:37:57.528524 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:37:57.528469    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:31.238488 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:31.238432    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:31.302031 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:31.301999    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:33.551736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:33.551703    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:33.651217 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:33.651167    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:33.719742 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:33.719702    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:34.457417 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:34.457382    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:43.255941 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:43.255903    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:43.356846 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:43.356816    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:44.050860 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:44.050831    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:44.149374 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:44.149335    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:44.229779 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:44.229748    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:47.623173 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:47.623143    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:47.723579 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:47.723548    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:47.823360 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:47.823328    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:38:58.980880 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:38:58.980799    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:39:22.427157 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:22.427122    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:39:34.458777 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:34.458747    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 18:39:34.461484 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:34.461464    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 18:39:34.463911 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:34.463890    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 18:39:45.851219 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:45.851176    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:39:46.790836 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.790806    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"]
Jul 29 18:39:46.791141 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.791123    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc56110-d7df-4d92-be67-fb6ee4f4ca30" containerName="registry"
Jul 29 18:39:46.791247 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.791142    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc56110-d7df-4d92-be67-fb6ee4f4ca30" containerName="registry"
Jul 29 18:39:46.791247 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.791233    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc56110-d7df-4d92-be67-fb6ee4f4ca30" containerName="registry"
Jul 29 18:39:46.792904 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.792884    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"
Jul 29 18:39:46.795342 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.795321    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 18:39:46.795430 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.795378    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-drbqn\""
Jul 29 18:39:46.796246 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.796232    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 18:39:46.808908 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.808871    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"]
Jul 29 18:39:46.890076 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.890042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ngxr\" (UniqueName: \"kubernetes.io/projected/8042710e-6aa6-49c3-8244-e4c46fe7204c-kube-api-access-4ngxr\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8\" (UID: \"8042710e-6aa6-49c3-8244-e4c46fe7204c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"
Jul 29 18:39:46.890076 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.890076    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8042710e-6aa6-49c3-8244-e4c46fe7204c-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8\" (UID: \"8042710e-6aa6-49c3-8244-e4c46fe7204c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"
Jul 29 18:39:46.890507 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.890101    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8042710e-6aa6-49c3-8244-e4c46fe7204c-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8\" (UID: \"8042710e-6aa6-49c3-8244-e4c46fe7204c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"
Jul 29 18:39:46.990787 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.990751    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ngxr\" (UniqueName: \"kubernetes.io/projected/8042710e-6aa6-49c3-8244-e4c46fe7204c-kube-api-access-4ngxr\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8\" (UID: \"8042710e-6aa6-49c3-8244-e4c46fe7204c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"
Jul 29 18:39:46.990787 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.990790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8042710e-6aa6-49c3-8244-e4c46fe7204c-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8\" (UID: \"8042710e-6aa6-49c3-8244-e4c46fe7204c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"
Jul 29 18:39:46.991018 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.990813    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8042710e-6aa6-49c3-8244-e4c46fe7204c-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8\" (UID: \"8042710e-6aa6-49c3-8244-e4c46fe7204c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"
Jul 29 18:39:46.991272 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.991248    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8042710e-6aa6-49c3-8244-e4c46fe7204c-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8\" (UID: \"8042710e-6aa6-49c3-8244-e4c46fe7204c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"
Jul 29 18:39:46.991351 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.991243    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8042710e-6aa6-49c3-8244-e4c46fe7204c-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8\" (UID: \"8042710e-6aa6-49c3-8244-e4c46fe7204c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"
Jul 29 18:39:46.999711 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:46.999688    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ngxr\" (UniqueName: \"kubernetes.io/projected/8042710e-6aa6-49c3-8244-e4c46fe7204c-kube-api-access-4ngxr\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8\" (UID: \"8042710e-6aa6-49c3-8244-e4c46fe7204c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"
Jul 29 18:39:47.102097 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:47.102065    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"
Jul 29 18:39:47.210846 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:47.210817    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:39:47.232150 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:47.232121    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"]
Jul 29 18:39:47.235172 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:39:47.235145    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8042710e_6aa6_49c3_8244_e4c46fe7204c.slice/crio-baed92944bb9b6df731427cf4cd0c63019b02b00deb7c5df00a53a50fe7f5c5f WatchSource:0}: Error finding container baed92944bb9b6df731427cf4cd0c63019b02b00deb7c5df00a53a50fe7f5c5f: Status 404 returned error can't find the container with id baed92944bb9b6df731427cf4cd0c63019b02b00deb7c5df00a53a50fe7f5c5f
Jul 29 18:39:47.237082 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:47.237066    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 18:39:47.866421 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:47.866385    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8" event={"ID":"8042710e-6aa6-49c3-8244-e4c46fe7204c","Type":"ContainerStarted","Data":"baed92944bb9b6df731427cf4cd0c63019b02b00deb7c5df00a53a50fe7f5c5f"}
Jul 29 18:39:53.888589 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:53.888558    2560 generic.go:356] "Generic (PLEG): container finished" podID="8042710e-6aa6-49c3-8244-e4c46fe7204c" containerID="bfe9237844e675f764e23859582023ea247f778cbb59590f67785124efc9d2a1" exitCode=0
Jul 29 18:39:53.888961 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:53.888648    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8" event={"ID":"8042710e-6aa6-49c3-8244-e4c46fe7204c","Type":"ContainerDied","Data":"bfe9237844e675f764e23859582023ea247f778cbb59590f67785124efc9d2a1"}
Jul 29 18:39:57.903867 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:57.903824    2560 generic.go:356] "Generic (PLEG): container finished" podID="8042710e-6aa6-49c3-8244-e4c46fe7204c" containerID="9b3426c903f19432f1d490e4652a21a7ddb6b0927a6ffb602a67261528ff2c42" exitCode=0
Jul 29 18:39:57.904257 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:39:57.903898    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8" event={"ID":"8042710e-6aa6-49c3-8244-e4c46fe7204c","Type":"ContainerDied","Data":"9b3426c903f19432f1d490e4652a21a7ddb6b0927a6ffb602a67261528ff2c42"}
Jul 29 18:40:04.930544 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:04.930462    2560 generic.go:356] "Generic (PLEG): container finished" podID="8042710e-6aa6-49c3-8244-e4c46fe7204c" containerID="caa29e53dabe4c8cec9e10544760ed5bbc40e9a6beac013d38bc480d30521565" exitCode=0
Jul 29 18:40:04.930897 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:04.930547    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8" event={"ID":"8042710e-6aa6-49c3-8244-e4c46fe7204c","Type":"ContainerDied","Data":"caa29e53dabe4c8cec9e10544760ed5bbc40e9a6beac013d38bc480d30521565"}
Jul 29 18:40:06.058410 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:06.058385    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"
Jul 29 18:40:06.236247 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:06.236135    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8042710e-6aa6-49c3-8244-e4c46fe7204c-bundle\") pod \"8042710e-6aa6-49c3-8244-e4c46fe7204c\" (UID: \"8042710e-6aa6-49c3-8244-e4c46fe7204c\") "
Jul 29 18:40:06.236247 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:06.236178    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8042710e-6aa6-49c3-8244-e4c46fe7204c-util\") pod \"8042710e-6aa6-49c3-8244-e4c46fe7204c\" (UID: \"8042710e-6aa6-49c3-8244-e4c46fe7204c\") "
Jul 29 18:40:06.236247 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:06.236235    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ngxr\" (UniqueName: \"kubernetes.io/projected/8042710e-6aa6-49c3-8244-e4c46fe7204c-kube-api-access-4ngxr\") pod \"8042710e-6aa6-49c3-8244-e4c46fe7204c\" (UID: \"8042710e-6aa6-49c3-8244-e4c46fe7204c\") "
Jul 29 18:40:06.236858 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:06.236827    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8042710e-6aa6-49c3-8244-e4c46fe7204c-bundle" (OuterVolumeSpecName: "bundle") pod "8042710e-6aa6-49c3-8244-e4c46fe7204c" (UID: "8042710e-6aa6-49c3-8244-e4c46fe7204c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:40:06.238371 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:06.238339    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8042710e-6aa6-49c3-8244-e4c46fe7204c-kube-api-access-4ngxr" (OuterVolumeSpecName: "kube-api-access-4ngxr") pod "8042710e-6aa6-49c3-8244-e4c46fe7204c" (UID: "8042710e-6aa6-49c3-8244-e4c46fe7204c"). InnerVolumeSpecName "kube-api-access-4ngxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:40:06.241258 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:06.241233    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8042710e-6aa6-49c3-8244-e4c46fe7204c-util" (OuterVolumeSpecName: "util") pod "8042710e-6aa6-49c3-8244-e4c46fe7204c" (UID: "8042710e-6aa6-49c3-8244-e4c46fe7204c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:40:06.337453 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:06.337421    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8042710e-6aa6-49c3-8244-e4c46fe7204c-bundle\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:40:06.337453 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:06.337447    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8042710e-6aa6-49c3-8244-e4c46fe7204c-util\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:40:06.337453 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:06.337457    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ngxr\" (UniqueName: \"kubernetes.io/projected/8042710e-6aa6-49c3-8244-e4c46fe7204c-kube-api-access-4ngxr\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:40:06.939351 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:06.939316    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8" event={"ID":"8042710e-6aa6-49c3-8244-e4c46fe7204c","Type":"ContainerDied","Data":"baed92944bb9b6df731427cf4cd0c63019b02b00deb7c5df00a53a50fe7f5c5f"}
Jul 29 18:40:06.939520 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:06.939367    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baed92944bb9b6df731427cf4cd0c63019b02b00deb7c5df00a53a50fe7f5c5f"
Jul 29 18:40:06.939520 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:06.939330    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pr2g8"
Jul 29 18:40:08.503585 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:08.503548    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:08.635233 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:08.635183    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:14.052864 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.052834    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:14.359419 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.359386    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-cgjgb"]
Jul 29 18:40:14.359679 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.359644    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8042710e-6aa6-49c3-8244-e4c46fe7204c" containerName="extract"
Jul 29 18:40:14.359679 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.359659    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8042710e-6aa6-49c3-8244-e4c46fe7204c" containerName="extract"
Jul 29 18:40:14.359679 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.359672    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8042710e-6aa6-49c3-8244-e4c46fe7204c" containerName="util"
Jul 29 18:40:14.359679 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.359677    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8042710e-6aa6-49c3-8244-e4c46fe7204c" containerName="util"
Jul 29 18:40:14.359803 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.359683    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8042710e-6aa6-49c3-8244-e4c46fe7204c" containerName="pull"
Jul 29 18:40:14.359803 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.359692    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8042710e-6aa6-49c3-8244-e4c46fe7204c" containerName="pull"
Jul 29 18:40:14.359803 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.359735    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8042710e-6aa6-49c3-8244-e4c46fe7204c" containerName="extract"
Jul 29 18:40:14.361576 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.361552    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:14.361671 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.361638    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-cgjgb"
Jul 29 18:40:14.364712 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.364687    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 29 18:40:14.364712 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.364687    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 29 18:40:14.365525 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.365503    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 29 18:40:14.365525 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.365514    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 29 18:40:14.365525 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.365521    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 29 18:40:14.365727 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.365504    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-tvzdv\""
Jul 29 18:40:14.380231 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.380193    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-cgjgb"]
Jul 29 18:40:14.476249 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.476220    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:14.489621 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.489590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl9qt\" (UniqueName: \"kubernetes.io/projected/cccd26d5-a088-463d-904e-9c449d108a12-kube-api-access-kl9qt\") pod \"keda-operator-77f648d694-cgjgb\" (UID: \"cccd26d5-a088-463d-904e-9c449d108a12\") " pod="openshift-keda/keda-operator-77f648d694-cgjgb"
Jul 29 18:40:14.489696 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.489652    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cccd26d5-a088-463d-904e-9c449d108a12-certificates\") pod \"keda-operator-77f648d694-cgjgb\" (UID: \"cccd26d5-a088-463d-904e-9c449d108a12\") " pod="openshift-keda/keda-operator-77f648d694-cgjgb"
Jul 29 18:40:14.489732 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.489716    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/cccd26d5-a088-463d-904e-9c449d108a12-cabundle0\") pod \"keda-operator-77f648d694-cgjgb\" (UID: \"cccd26d5-a088-463d-904e-9c449d108a12\") " pod="openshift-keda/keda-operator-77f648d694-cgjgb"
Jul 29 18:40:14.590069 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.590034    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cccd26d5-a088-463d-904e-9c449d108a12-certificates\") pod \"keda-operator-77f648d694-cgjgb\" (UID: \"cccd26d5-a088-463d-904e-9c449d108a12\") " pod="openshift-keda/keda-operator-77f648d694-cgjgb"
Jul 29 18:40:14.590263 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.590092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/cccd26d5-a088-463d-904e-9c449d108a12-cabundle0\") pod \"keda-operator-77f648d694-cgjgb\" (UID: \"cccd26d5-a088-463d-904e-9c449d108a12\") " pod="openshift-keda/keda-operator-77f648d694-cgjgb"
Jul 29 18:40:14.590263 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.590132    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl9qt\" (UniqueName: \"kubernetes.io/projected/cccd26d5-a088-463d-904e-9c449d108a12-kube-api-access-kl9qt\") pod \"keda-operator-77f648d694-cgjgb\" (UID: \"cccd26d5-a088-463d-904e-9c449d108a12\") " pod="openshift-keda/keda-operator-77f648d694-cgjgb"
Jul 29 18:40:14.590263 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:40:14.590147    2560 secret.go:281] references non-existent secret key: ca.crt
Jul 29 18:40:14.590263 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:40:14.590163    2560 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 29 18:40:14.590263 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:40:14.590175    2560 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-cgjgb: references non-existent secret key: ca.crt
Jul 29 18:40:14.590263 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:40:14.590252    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cccd26d5-a088-463d-904e-9c449d108a12-certificates podName:cccd26d5-a088-463d-904e-9c449d108a12 nodeName:}" failed. No retries permitted until 2026-07-29 18:40:15.090232365 +0000 UTC m=+340.984232767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/cccd26d5-a088-463d-904e-9c449d108a12-certificates") pod "keda-operator-77f648d694-cgjgb" (UID: "cccd26d5-a088-463d-904e-9c449d108a12") : references non-existent secret key: ca.crt
Jul 29 18:40:14.590874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.590852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/cccd26d5-a088-463d-904e-9c449d108a12-cabundle0\") pod \"keda-operator-77f648d694-cgjgb\" (UID: \"cccd26d5-a088-463d-904e-9c449d108a12\") " pod="openshift-keda/keda-operator-77f648d694-cgjgb"
Jul 29 18:40:14.598806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.598772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl9qt\" (UniqueName: \"kubernetes.io/projected/cccd26d5-a088-463d-904e-9c449d108a12-kube-api-access-kl9qt\") pod \"keda-operator-77f648d694-cgjgb\" (UID: \"cccd26d5-a088-463d-904e-9c449d108a12\") " pod="openshift-keda/keda-operator-77f648d694-cgjgb"
Jul 29 18:40:14.896421 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:14.896388    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:15.003768 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.003735    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"]
Jul 29 18:40:15.006505 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.006475    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"
Jul 29 18:40:15.009258 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.009214    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 29 18:40:15.016940 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.016916    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"]
Jul 29 18:40:15.095013 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.094974    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhss\" (UniqueName: \"kubernetes.io/projected/53599807-f832-410e-9708-77f758d67c96-kube-api-access-plhss\") pod \"keda-metrics-apiserver-55557df8c-2sgls\" (UID: \"53599807-f832-410e-9708-77f758d67c96\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"
Jul 29 18:40:15.095013 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.095015    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/53599807-f832-410e-9708-77f758d67c96-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-2sgls\" (UID: \"53599807-f832-410e-9708-77f758d67c96\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"
Jul 29 18:40:15.095426 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.095038    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/53599807-f832-410e-9708-77f758d67c96-certificates\") pod \"keda-metrics-apiserver-55557df8c-2sgls\" (UID: \"53599807-f832-410e-9708-77f758d67c96\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"
Jul 29 18:40:15.095426 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.095085    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cccd26d5-a088-463d-904e-9c449d108a12-certificates\") pod \"keda-operator-77f648d694-cgjgb\" (UID: \"cccd26d5-a088-463d-904e-9c449d108a12\") " pod="openshift-keda/keda-operator-77f648d694-cgjgb"
Jul 29 18:40:15.097410 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.097390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cccd26d5-a088-463d-904e-9c449d108a12-certificates\") pod \"keda-operator-77f648d694-cgjgb\" (UID: \"cccd26d5-a088-463d-904e-9c449d108a12\") " pod="openshift-keda/keda-operator-77f648d694-cgjgb"
Jul 29 18:40:15.116861 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.116834    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:15.195800 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.195720    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/53599807-f832-410e-9708-77f758d67c96-certificates\") pod \"keda-metrics-apiserver-55557df8c-2sgls\" (UID: \"53599807-f832-410e-9708-77f758d67c96\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"
Jul 29 18:40:15.195800 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.195787    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhss\" (UniqueName: \"kubernetes.io/projected/53599807-f832-410e-9708-77f758d67c96-kube-api-access-plhss\") pod \"keda-metrics-apiserver-55557df8c-2sgls\" (UID: \"53599807-f832-410e-9708-77f758d67c96\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"
Jul 29 18:40:15.195988 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.195811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/53599807-f832-410e-9708-77f758d67c96-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-2sgls\" (UID: \"53599807-f832-410e-9708-77f758d67c96\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"
Jul 29 18:40:15.195988 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:40:15.195870    2560 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 29 18:40:15.195988 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:40:15.195894    2560 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-2sgls: secret "keda-metrics-apiserver-certs" not found
Jul 29 18:40:15.195988 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:40:15.195947    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53599807-f832-410e-9708-77f758d67c96-certificates podName:53599807-f832-410e-9708-77f758d67c96 nodeName:}" failed. No retries permitted until 2026-07-29 18:40:15.695931398 +0000 UTC m=+341.589931777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/53599807-f832-410e-9708-77f758d67c96-certificates") pod "keda-metrics-apiserver-55557df8c-2sgls" (UID: "53599807-f832-410e-9708-77f758d67c96") : secret "keda-metrics-apiserver-certs" not found
Jul 29 18:40:15.196174 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.196156    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/53599807-f832-410e-9708-77f758d67c96-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-2sgls\" (UID: \"53599807-f832-410e-9708-77f758d67c96\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"
Jul 29 18:40:15.203933 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.203915    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhss\" (UniqueName: \"kubernetes.io/projected/53599807-f832-410e-9708-77f758d67c96-kube-api-access-plhss\") pod \"keda-metrics-apiserver-55557df8c-2sgls\" (UID: \"53599807-f832-410e-9708-77f758d67c96\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"
Jul 29 18:40:15.216367 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.216341    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:15.271402 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.271373    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-cgjgb"
Jul 29 18:40:15.314875 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.314843    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:15.397801 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.397770    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-cgjgb"]
Jul 29 18:40:15.400637 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:40:15.400612    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccd26d5_a088_463d_904e_9c449d108a12.slice/crio-5962644b726fe14b89aa86a690ad566a49b3c6224c9bd98cecc7b4aa131f0d9e WatchSource:0}: Error finding container 5962644b726fe14b89aa86a690ad566a49b3c6224c9bd98cecc7b4aa131f0d9e: Status 404 returned error can't find the container with id 5962644b726fe14b89aa86a690ad566a49b3c6224c9bd98cecc7b4aa131f0d9e
Jul 29 18:40:15.426604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.426554    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:15.682382 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.682352    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:15.699525 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.699482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/53599807-f832-410e-9708-77f758d67c96-certificates\") pod \"keda-metrics-apiserver-55557df8c-2sgls\" (UID: \"53599807-f832-410e-9708-77f758d67c96\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"
Jul 29 18:40:15.702035 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.702013    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/53599807-f832-410e-9708-77f758d67c96-certificates\") pod \"keda-metrics-apiserver-55557df8c-2sgls\" (UID: \"53599807-f832-410e-9708-77f758d67c96\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"
Jul 29 18:40:15.883744 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.883694    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:15.918959 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.918920    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"
Jul 29 18:40:15.972264 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.972234    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-cgjgb" event={"ID":"cccd26d5-a088-463d-904e-9c449d108a12","Type":"ContainerStarted","Data":"5962644b726fe14b89aa86a690ad566a49b3c6224c9bd98cecc7b4aa131f0d9e"}
Jul 29 18:40:15.985152 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:15.985125    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:16.043838 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:16.043806    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"]
Jul 29 18:40:16.046968 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:40:16.046938    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53599807_f832_410e_9708_77f758d67c96.slice/crio-56a33a5bde707c294290daa0af7e342467678e791cc4d0f1d7b0bb260daf7263 WatchSource:0}: Error finding container 56a33a5bde707c294290daa0af7e342467678e791cc4d0f1d7b0bb260daf7263: Status 404 returned error can't find the container with id 56a33a5bde707c294290daa0af7e342467678e791cc4d0f1d7b0bb260daf7263
Jul 29 18:40:16.978371 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:16.978331    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls" event={"ID":"53599807-f832-410e-9708-77f758d67c96","Type":"ContainerStarted","Data":"56a33a5bde707c294290daa0af7e342467678e791cc4d0f1d7b0bb260daf7263"}
Jul 29 18:40:19.990514 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:19.990479    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-cgjgb" event={"ID":"cccd26d5-a088-463d-904e-9c449d108a12","Type":"ContainerStarted","Data":"c827732c4692fcae5f4407dd72e9c95bc0d3213f27cf4b8a1344f21729956897"}
Jul 29 18:40:19.990954 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:19.990544    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-cgjgb"
Jul 29 18:40:19.991946 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:19.991924    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls" event={"ID":"53599807-f832-410e-9708-77f758d67c96","Type":"ContainerStarted","Data":"4258716318b39b146baa702bce52686b5d167ee5c980fc0a8d175d1519bd1c81"}
Jul 29 18:40:19.992076 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:19.992045    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"
Jul 29 18:40:20.010892 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:20.010848    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-cgjgb" podStartSLOduration=2.182324144 podStartE2EDuration="6.010833318s" totalImagesPullingTime="3.828509174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:20.009418218 +0000 UTC m=+345.903418620" watchObservedRunningTime="2026-07-29 18:40:20.010833318 +0000 UTC m=+345.904833721"
Jul 29 18:40:20.033582 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:20.033538    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls" podStartSLOduration=2.852420995 podStartE2EDuration="6.033525312s" totalImagesPullingTime="3.181104317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:20.033187937 +0000 UTC m=+345.927188340" watchObservedRunningTime="2026-07-29 18:40:20.033525312 +0000 UTC m=+345.927525713"
Jul 29 18:40:30.999472 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:30.999389    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-2sgls"
Jul 29 18:40:36.004522 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:36.004490    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:36.097921 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:36.097885    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:40:40.997243 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:40.997208    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-cgjgb"
Jul 29 18:40:59.163528 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:40:59.163492    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:01.077074 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:01.077042    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:01.140842 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:01.140811    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:07.214862 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.214829    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:07.315924 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.315891    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:07.466227 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.466136    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"]
Jul 29 18:41:07.468334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.468316    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"
Jul 29 18:41:07.470963 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.470944    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 18:41:07.471060 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.470944    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 18:41:07.471807 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.471786    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-drbqn\""
Jul 29 18:41:07.478078 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.478053    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"]
Jul 29 18:41:07.563002 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.562965    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c406020-7801-44d7-a58d-5318e7ce43d5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq\" (UID: \"2c406020-7801-44d7-a58d-5318e7ce43d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"
Jul 29 18:41:07.563168 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.563014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c406020-7801-44d7-a58d-5318e7ce43d5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq\" (UID: \"2c406020-7801-44d7-a58d-5318e7ce43d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"
Jul 29 18:41:07.563168 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.563058    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bg7c\" (UniqueName: \"kubernetes.io/projected/2c406020-7801-44d7-a58d-5318e7ce43d5-kube-api-access-6bg7c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq\" (UID: \"2c406020-7801-44d7-a58d-5318e7ce43d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"
Jul 29 18:41:07.663996 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.663962    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c406020-7801-44d7-a58d-5318e7ce43d5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq\" (UID: \"2c406020-7801-44d7-a58d-5318e7ce43d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"
Jul 29 18:41:07.664157 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.664013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c406020-7801-44d7-a58d-5318e7ce43d5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq\" (UID: \"2c406020-7801-44d7-a58d-5318e7ce43d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"
Jul 29 18:41:07.664157 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.664047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bg7c\" (UniqueName: \"kubernetes.io/projected/2c406020-7801-44d7-a58d-5318e7ce43d5-kube-api-access-6bg7c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq\" (UID: \"2c406020-7801-44d7-a58d-5318e7ce43d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"
Jul 29 18:41:07.664455 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.664435    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c406020-7801-44d7-a58d-5318e7ce43d5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq\" (UID: \"2c406020-7801-44d7-a58d-5318e7ce43d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"
Jul 29 18:41:07.664495 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.664475    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c406020-7801-44d7-a58d-5318e7ce43d5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq\" (UID: \"2c406020-7801-44d7-a58d-5318e7ce43d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"
Jul 29 18:41:07.672383 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.672363    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bg7c\" (UniqueName: \"kubernetes.io/projected/2c406020-7801-44d7-a58d-5318e7ce43d5-kube-api-access-6bg7c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq\" (UID: \"2c406020-7801-44d7-a58d-5318e7ce43d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"
Jul 29 18:41:07.777567 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.777480    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"
Jul 29 18:41:07.901784 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:07.901759    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"]
Jul 29 18:41:07.904028 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:41:07.903995    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c406020_7801_44d7_a58d_5318e7ce43d5.slice/crio-a78c839e27a4bfc69052078987973a8b05cb0786430dfc53fa706da5e6a5e745 WatchSource:0}: Error finding container a78c839e27a4bfc69052078987973a8b05cb0786430dfc53fa706da5e6a5e745: Status 404 returned error can't find the container with id a78c839e27a4bfc69052078987973a8b05cb0786430dfc53fa706da5e6a5e745
Jul 29 18:41:08.161565 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:08.161530    2560 generic.go:356] "Generic (PLEG): container finished" podID="2c406020-7801-44d7-a58d-5318e7ce43d5" containerID="5ac99ebcebcee02bfe633a836fe926d44ea1c0637cdbf861c33a61a3a527a53d" exitCode=0
Jul 29 18:41:08.161742 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:08.161588    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq" event={"ID":"2c406020-7801-44d7-a58d-5318e7ce43d5","Type":"ContainerDied","Data":"5ac99ebcebcee02bfe633a836fe926d44ea1c0637cdbf861c33a61a3a527a53d"}
Jul 29 18:41:08.161742 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:08.161633    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq" event={"ID":"2c406020-7801-44d7-a58d-5318e7ce43d5","Type":"ContainerStarted","Data":"a78c839e27a4bfc69052078987973a8b05cb0786430dfc53fa706da5e6a5e745"}
Jul 29 18:41:10.171025 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:10.170993    2560 generic.go:356] "Generic (PLEG): container finished" podID="2c406020-7801-44d7-a58d-5318e7ce43d5" containerID="095a01b0b78474824440c9a9426c1b8cedcf53726219df17a59a46d4b4a91a19" exitCode=0
Jul 29 18:41:10.171422 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:10.171047    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq" event={"ID":"2c406020-7801-44d7-a58d-5318e7ce43d5","Type":"ContainerDied","Data":"095a01b0b78474824440c9a9426c1b8cedcf53726219df17a59a46d4b4a91a19"}
Jul 29 18:41:11.176723 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:11.176691    2560 generic.go:356] "Generic (PLEG): container finished" podID="2c406020-7801-44d7-a58d-5318e7ce43d5" containerID="66b9d3a301efaa661940339c8c26e082b19023ffc448b7d6e08dada33c0ed206" exitCode=0
Jul 29 18:41:11.177095 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:11.176757    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq" event={"ID":"2c406020-7801-44d7-a58d-5318e7ce43d5","Type":"ContainerDied","Data":"66b9d3a301efaa661940339c8c26e082b19023ffc448b7d6e08dada33c0ed206"}
Jul 29 18:41:12.311095 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:12.311066    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"
Jul 29 18:41:12.404427 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:12.404391    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bg7c\" (UniqueName: \"kubernetes.io/projected/2c406020-7801-44d7-a58d-5318e7ce43d5-kube-api-access-6bg7c\") pod \"2c406020-7801-44d7-a58d-5318e7ce43d5\" (UID: \"2c406020-7801-44d7-a58d-5318e7ce43d5\") "
Jul 29 18:41:12.404582 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:12.404463    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c406020-7801-44d7-a58d-5318e7ce43d5-util\") pod \"2c406020-7801-44d7-a58d-5318e7ce43d5\" (UID: \"2c406020-7801-44d7-a58d-5318e7ce43d5\") "
Jul 29 18:41:12.404582 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:12.404490    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c406020-7801-44d7-a58d-5318e7ce43d5-bundle\") pod \"2c406020-7801-44d7-a58d-5318e7ce43d5\" (UID: \"2c406020-7801-44d7-a58d-5318e7ce43d5\") "
Jul 29 18:41:12.405184 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:12.405156    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c406020-7801-44d7-a58d-5318e7ce43d5-bundle" (OuterVolumeSpecName: "bundle") pod "2c406020-7801-44d7-a58d-5318e7ce43d5" (UID: "2c406020-7801-44d7-a58d-5318e7ce43d5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:41:12.406428 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:12.406399    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c406020-7801-44d7-a58d-5318e7ce43d5-kube-api-access-6bg7c" (OuterVolumeSpecName: "kube-api-access-6bg7c") pod "2c406020-7801-44d7-a58d-5318e7ce43d5" (UID: "2c406020-7801-44d7-a58d-5318e7ce43d5"). InnerVolumeSpecName "kube-api-access-6bg7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:41:12.409895 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:12.409857    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c406020-7801-44d7-a58d-5318e7ce43d5-util" (OuterVolumeSpecName: "util") pod "2c406020-7801-44d7-a58d-5318e7ce43d5" (UID: "2c406020-7801-44d7-a58d-5318e7ce43d5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:41:12.505389 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:12.505311    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c406020-7801-44d7-a58d-5318e7ce43d5-util\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:41:12.505389 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:12.505338    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c406020-7801-44d7-a58d-5318e7ce43d5-bundle\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:41:12.505389 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:12.505348    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bg7c\" (UniqueName: \"kubernetes.io/projected/2c406020-7801-44d7-a58d-5318e7ce43d5-kube-api-access-6bg7c\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:41:13.186451 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:13.186417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq" event={"ID":"2c406020-7801-44d7-a58d-5318e7ce43d5","Type":"ContainerDied","Data":"a78c839e27a4bfc69052078987973a8b05cb0786430dfc53fa706da5e6a5e745"}
Jul 29 18:41:13.186591 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:13.186468    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a78c839e27a4bfc69052078987973a8b05cb0786430dfc53fa706da5e6a5e745"
Jul 29 18:41:13.186591 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:13.186427    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sw6tq"
Jul 29 18:41:18.391838 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:18.391801    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:19.134078 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:19.134043    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:20.449285 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.449251    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-48msr"]
Jul 29 18:41:20.449649 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.449493    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c406020-7801-44d7-a58d-5318e7ce43d5" containerName="extract"
Jul 29 18:41:20.449649 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.449505    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c406020-7801-44d7-a58d-5318e7ce43d5" containerName="extract"
Jul 29 18:41:20.449649 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.449520    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c406020-7801-44d7-a58d-5318e7ce43d5" containerName="util"
Jul 29 18:41:20.449649 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.449526    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c406020-7801-44d7-a58d-5318e7ce43d5" containerName="util"
Jul 29 18:41:20.449649 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.449538    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c406020-7801-44d7-a58d-5318e7ce43d5" containerName="pull"
Jul 29 18:41:20.449649 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.449544    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c406020-7801-44d7-a58d-5318e7ce43d5" containerName="pull"
Jul 29 18:41:20.449649 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.449591    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c406020-7801-44d7-a58d-5318e7ce43d5" containerName="extract"
Jul 29 18:41:20.451511 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.451493    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-48msr"
Jul 29 18:41:20.454033 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.454011    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 29 18:41:20.454033 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.454013    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 29 18:41:20.454236 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.454074    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-s2rr2\""
Jul 29 18:41:20.467871 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.467848    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-48msr"]
Jul 29 18:41:20.561299 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.561266    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf955ff7-211c-4804-8b30-6ac32b4bb7e2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-48msr\" (UID: \"bf955ff7-211c-4804-8b30-6ac32b4bb7e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-48msr"
Jul 29 18:41:20.561473 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.561312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5cz\" (UniqueName: \"kubernetes.io/projected/bf955ff7-211c-4804-8b30-6ac32b4bb7e2-kube-api-access-lm5cz\") pod \"cert-manager-operator-controller-manager-55c444bf86-48msr\" (UID: \"bf955ff7-211c-4804-8b30-6ac32b4bb7e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-48msr"
Jul 29 18:41:20.662487 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.662440    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf955ff7-211c-4804-8b30-6ac32b4bb7e2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-48msr\" (UID: \"bf955ff7-211c-4804-8b30-6ac32b4bb7e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-48msr"
Jul 29 18:41:20.662487 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.662498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5cz\" (UniqueName: \"kubernetes.io/projected/bf955ff7-211c-4804-8b30-6ac32b4bb7e2-kube-api-access-lm5cz\") pod \"cert-manager-operator-controller-manager-55c444bf86-48msr\" (UID: \"bf955ff7-211c-4804-8b30-6ac32b4bb7e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-48msr"
Jul 29 18:41:20.662900 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.662876    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf955ff7-211c-4804-8b30-6ac32b4bb7e2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-48msr\" (UID: \"bf955ff7-211c-4804-8b30-6ac32b4bb7e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-48msr"
Jul 29 18:41:20.674711 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.674681    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5cz\" (UniqueName: \"kubernetes.io/projected/bf955ff7-211c-4804-8b30-6ac32b4bb7e2-kube-api-access-lm5cz\") pod \"cert-manager-operator-controller-manager-55c444bf86-48msr\" (UID: \"bf955ff7-211c-4804-8b30-6ac32b4bb7e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-48msr"
Jul 29 18:41:20.760253 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.760135    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-48msr"
Jul 29 18:41:20.895760 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:20.895573    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-48msr"]
Jul 29 18:41:20.898643 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:41:20.898619    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf955ff7_211c_4804_8b30_6ac32b4bb7e2.slice/crio-8a8be58f4cdde3e6830efb2f57009ac90199facf5ba98f8a351bc2640050f3bd WatchSource:0}: Error finding container 8a8be58f4cdde3e6830efb2f57009ac90199facf5ba98f8a351bc2640050f3bd: Status 404 returned error can't find the container with id 8a8be58f4cdde3e6830efb2f57009ac90199facf5ba98f8a351bc2640050f3bd
Jul 29 18:41:21.216414 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:21.216379    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-48msr" event={"ID":"bf955ff7-211c-4804-8b30-6ac32b4bb7e2","Type":"ContainerStarted","Data":"8a8be58f4cdde3e6830efb2f57009ac90199facf5ba98f8a351bc2640050f3bd"}
Jul 29 18:41:21.984805 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:21.984756    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:22.085062 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:22.085032    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:23.226993 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:23.226951    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-48msr" event={"ID":"bf955ff7-211c-4804-8b30-6ac32b4bb7e2","Type":"ContainerStarted","Data":"0553f4e95f1263219b2ccfc8773bb83f592a8aa892b27eb199f2da5308ff6af0"}
Jul 29 18:41:23.251278 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:23.251214    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-48msr" podStartSLOduration=1.636992088 podStartE2EDuration="3.251177967s" totalImagesPullingTime="1.614185879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:41:23.248941301 +0000 UTC m=+409.142941703" watchObservedRunningTime="2026-07-29 18:41:23.251177967 +0000 UTC m=+409.145178370"
Jul 29 18:41:24.291217 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:24.291159    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:24.379396 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:24.379367    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:26.503672 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:26.503642    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:26.688459 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:26.688427    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:26.974121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:26.974085    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:27.074575 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:27.074547    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:28.814249 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:28.814180    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:28.917844 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:28.917809    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:29.015359 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:29.015329    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:29.986723 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:29.986682    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"]
Jul 29 18:41:29.989685 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:29.989667    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"
Jul 29 18:41:29.992502 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:29.992481    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 18:41:29.992645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:29.992629    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 18:41:29.993437 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:29.993417    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-drbqn\""
Jul 29 18:41:29.997961 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:29.997936    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"]
Jul 29 18:41:30.134415 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:30.134377    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb\" (UID: \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"
Jul 29 18:41:30.134415 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:30.134424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb\" (UID: \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"
Jul 29 18:41:30.134635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:30.134505    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvvng\" (UniqueName: \"kubernetes.io/projected/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-kube-api-access-cvvng\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb\" (UID: \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"
Jul 29 18:41:30.235211 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:30.235159    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb\" (UID: \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"
Jul 29 18:41:30.235445 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:30.235225    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb\" (UID: \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"
Jul 29 18:41:30.235445 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:30.235269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvvng\" (UniqueName: \"kubernetes.io/projected/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-kube-api-access-cvvng\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb\" (UID: \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"
Jul 29 18:41:30.235585 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:30.235566    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb\" (UID: \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"
Jul 29 18:41:30.235623 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:30.235599    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb\" (UID: \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"
Jul 29 18:41:30.245177 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:30.245118    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvvng\" (UniqueName: \"kubernetes.io/projected/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-kube-api-access-cvvng\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb\" (UID: \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"
Jul 29 18:41:30.299760 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:30.299725    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"
Jul 29 18:41:30.444900 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:30.444871    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"]
Jul 29 18:41:30.447384 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:41:30.447355    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5b0ed8_4ba4_41f2_bd4f_be859485fd42.slice/crio-13680a07277b25c49e2738987711fc8cc06f62ebe78a797d2cc920e71c8af2fc WatchSource:0}: Error finding container 13680a07277b25c49e2738987711fc8cc06f62ebe78a797d2cc920e71c8af2fc: Status 404 returned error can't find the container with id 13680a07277b25c49e2738987711fc8cc06f62ebe78a797d2cc920e71c8af2fc
Jul 29 18:41:31.258973 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:31.258942    2560 generic.go:356] "Generic (PLEG): container finished" podID="dd5b0ed8-4ba4-41f2-bd4f-be859485fd42" containerID="a78a7847273d3d15da395df696f5b308cf9b17f06a0413270b0a40c81ee5caa2" exitCode=0
Jul 29 18:41:31.259345 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:31.259031    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb" event={"ID":"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42","Type":"ContainerDied","Data":"a78a7847273d3d15da395df696f5b308cf9b17f06a0413270b0a40c81ee5caa2"}
Jul 29 18:41:31.259345 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:31.259093    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb" event={"ID":"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42","Type":"ContainerStarted","Data":"13680a07277b25c49e2738987711fc8cc06f62ebe78a797d2cc920e71c8af2fc"}
Jul 29 18:41:32.403590 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:32.403555    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-65rjd"]
Jul 29 18:41:32.406868 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:32.406846    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-65rjd"
Jul 29 18:41:32.409757 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:32.409729    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 29 18:41:32.409891 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:32.409836    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 29 18:41:32.409955 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:32.409913    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jv86g\""
Jul 29 18:41:32.415033 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:32.414689    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-65rjd"]
Jul 29 18:41:32.561006 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:32.560971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5hpw\" (UniqueName: \"kubernetes.io/projected/36d597ea-1048-41b1-9b4d-fa0b12b1fbc6-kube-api-access-x5hpw\") pod \"cert-manager-webhook-6486ffd997-65rjd\" (UID: \"36d597ea-1048-41b1-9b4d-fa0b12b1fbc6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-65rjd"
Jul 29 18:41:32.561181 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:32.561019    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36d597ea-1048-41b1-9b4d-fa0b12b1fbc6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-65rjd\" (UID: \"36d597ea-1048-41b1-9b4d-fa0b12b1fbc6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-65rjd"
Jul 29 18:41:32.662389 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:32.662295    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36d597ea-1048-41b1-9b4d-fa0b12b1fbc6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-65rjd\" (UID: \"36d597ea-1048-41b1-9b4d-fa0b12b1fbc6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-65rjd"
Jul 29 18:41:32.662560 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:32.662417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5hpw\" (UniqueName: \"kubernetes.io/projected/36d597ea-1048-41b1-9b4d-fa0b12b1fbc6-kube-api-access-x5hpw\") pod \"cert-manager-webhook-6486ffd997-65rjd\" (UID: \"36d597ea-1048-41b1-9b4d-fa0b12b1fbc6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-65rjd"
Jul 29 18:41:32.671727 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:32.671698    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36d597ea-1048-41b1-9b4d-fa0b12b1fbc6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-65rjd\" (UID: \"36d597ea-1048-41b1-9b4d-fa0b12b1fbc6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-65rjd"
Jul 29 18:41:32.671918 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:32.671897    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5hpw\" (UniqueName: \"kubernetes.io/projected/36d597ea-1048-41b1-9b4d-fa0b12b1fbc6-kube-api-access-x5hpw\") pod \"cert-manager-webhook-6486ffd997-65rjd\" (UID: \"36d597ea-1048-41b1-9b4d-fa0b12b1fbc6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-65rjd"
Jul 29 18:41:32.718690 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:32.718648    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-65rjd"
Jul 29 18:41:33.300642 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:33.300588    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-65rjd"]
Jul 29 18:41:33.303391 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:41:33.303358    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36d597ea_1048_41b1_9b4d_fa0b12b1fbc6.slice/crio-cf4d9850a884db2ac63d85b13d4052f948158e9c9816f34cb098f64483a50cdd WatchSource:0}: Error finding container cf4d9850a884db2ac63d85b13d4052f948158e9c9816f34cb098f64483a50cdd: Status 404 returned error can't find the container with id cf4d9850a884db2ac63d85b13d4052f948158e9c9816f34cb098f64483a50cdd
Jul 29 18:41:34.275192 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:34.275143    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-65rjd" event={"ID":"36d597ea-1048-41b1-9b4d-fa0b12b1fbc6","Type":"ContainerStarted","Data":"cf4d9850a884db2ac63d85b13d4052f948158e9c9816f34cb098f64483a50cdd"}
Jul 29 18:41:34.277171 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:34.277132    2560 generic.go:356] "Generic (PLEG): container finished" podID="dd5b0ed8-4ba4-41f2-bd4f-be859485fd42" containerID="bef1f090b98f1f0b5b3e03eaddf2fade45a7b4b5e378650a746da146f9371071" exitCode=0
Jul 29 18:41:34.277316 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:34.277219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb" event={"ID":"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42","Type":"ContainerDied","Data":"bef1f090b98f1f0b5b3e03eaddf2fade45a7b4b5e378650a746da146f9371071"}
Jul 29 18:41:34.701467 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:34.701430    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:35.284127 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:35.284092    2560 generic.go:356] "Generic (PLEG): container finished" podID="dd5b0ed8-4ba4-41f2-bd4f-be859485fd42" containerID="2b6a8c6c9844f5fbbb66085320a0fe738ffb63ccc2c138b187d2bfc6aeb54912" exitCode=0
Jul 29 18:41:35.284598 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:35.284167    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb" event={"ID":"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42","Type":"ContainerDied","Data":"2b6a8c6c9844f5fbbb66085320a0fe738ffb63ccc2c138b187d2bfc6aeb54912"}
Jul 29 18:41:36.290347 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.290307    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-65rjd" event={"ID":"36d597ea-1048-41b1-9b4d-fa0b12b1fbc6","Type":"ContainerStarted","Data":"a35502f94de800ae770fcd80c42e148f502e2416b96f71a37996d221da06e810"}
Jul 29 18:41:36.290763 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.290492    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-65rjd"
Jul 29 18:41:36.313792 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.313730    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-65rjd" podStartSLOduration=1.5319536139999999 podStartE2EDuration="4.313708236s" totalImagesPullingTime="2.781754622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:41:36.311954232 +0000 UTC m=+422.205954634" watchObservedRunningTime="2026-07-29 18:41:36.313708236 +0000 UTC m=+422.207708638"
Jul 29 18:41:36.341920 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.341876    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:36.413046 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.413018    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"
Jul 29 18:41:36.435708 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.435679    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:36.590698 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.590672    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvvng\" (UniqueName: \"kubernetes.io/projected/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-kube-api-access-cvvng\") pod \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\" (UID: \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\") "
Jul 29 18:41:36.590890 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.590744    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-util\") pod \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\" (UID: \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\") "
Jul 29 18:41:36.590890 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.590801    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-bundle\") pod \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\" (UID: \"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42\") "
Jul 29 18:41:36.591213 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.591164    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-bundle" (OuterVolumeSpecName: "bundle") pod "dd5b0ed8-4ba4-41f2-bd4f-be859485fd42" (UID: "dd5b0ed8-4ba4-41f2-bd4f-be859485fd42"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:41:36.592736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.592707    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-kube-api-access-cvvng" (OuterVolumeSpecName: "kube-api-access-cvvng") pod "dd5b0ed8-4ba4-41f2-bd4f-be859485fd42" (UID: "dd5b0ed8-4ba4-41f2-bd4f-be859485fd42"). InnerVolumeSpecName "kube-api-access-cvvng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:41:36.594842 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.594820    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-util" (OuterVolumeSpecName: "util") pod "dd5b0ed8-4ba4-41f2-bd4f-be859485fd42" (UID: "dd5b0ed8-4ba4-41f2-bd4f-be859485fd42"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:41:36.691599 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.691568    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-bundle\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:41:36.691599 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.691593    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvvng\" (UniqueName: \"kubernetes.io/projected/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-kube-api-access-cvvng\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:41:36.691599 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:36.691602    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5b0ed8-4ba4-41f2-bd4f-be859485fd42-util\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:41:37.297121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:37.297073    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb" event={"ID":"dd5b0ed8-4ba4-41f2-bd4f-be859485fd42","Type":"ContainerDied","Data":"13680a07277b25c49e2738987711fc8cc06f62ebe78a797d2cc920e71c8af2fc"}
Jul 29 18:41:37.297513 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:37.297143    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13680a07277b25c49e2738987711fc8cc06f62ebe78a797d2cc920e71c8af2fc"
Jul 29 18:41:37.297513 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:37.297096    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb2kb"
Jul 29 18:41:40.598586 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:40.598552    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:40.699887 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:40.699805    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:42.299615 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:42.299584    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-65rjd"
Jul 29 18:41:52.366956 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:52.366926    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:57.088634 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.088554    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"]
Jul 29 18:41:57.088992 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.088804    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5b0ed8-4ba4-41f2-bd4f-be859485fd42" containerName="pull"
Jul 29 18:41:57.088992 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.088813    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5b0ed8-4ba4-41f2-bd4f-be859485fd42" containerName="pull"
Jul 29 18:41:57.088992 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.088831    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5b0ed8-4ba4-41f2-bd4f-be859485fd42" containerName="extract"
Jul 29 18:41:57.088992 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.088837    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5b0ed8-4ba4-41f2-bd4f-be859485fd42" containerName="extract"
Jul 29 18:41:57.088992 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.088847    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5b0ed8-4ba4-41f2-bd4f-be859485fd42" containerName="util"
Jul 29 18:41:57.088992 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.088853    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5b0ed8-4ba4-41f2-bd4f-be859485fd42" containerName="util"
Jul 29 18:41:57.088992 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.088893    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd5b0ed8-4ba4-41f2-bd4f-be859485fd42" containerName="extract"
Jul 29 18:41:57.090534 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.090517    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.093035 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.093011    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 29 18:41:57.093189 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.093017    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 29 18:41:57.093189 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.093074    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 18:41:57.093363 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.093346    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 18:41:57.093434 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.093352    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-94q2z\""
Jul 29 18:41:57.093434 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.093366    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 18:41:57.093434 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.093400    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 29 18:41:57.094716 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.094696    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:57.104813 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.104789    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"]
Jul 29 18:41:57.160313 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.160284    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:41:57.245387 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.245339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.245542 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.245404    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.245542 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.245420    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.245542 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.245439    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.245542 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.245533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.245701 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.245556    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.245701 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.245575    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-979w8\" (UniqueName: \"kubernetes.io/projected/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-kube-api-access-979w8\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.346363 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.346276    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.346363 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.346326    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.346363 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.346344    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.346363 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.346371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.346663 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.346421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.346663 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.346446    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.346663 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.346473    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-979w8\" (UniqueName: \"kubernetes.io/projected/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-kube-api-access-979w8\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.347149 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.347125    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.348780 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.348752    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.348856 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.348750    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.348960 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.348942    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.349071 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.349057    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.358590 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.358566    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-979w8\" (UniqueName: \"kubernetes.io/projected/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-kube-api-access-979w8\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.359690 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.359670    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-g8vpg\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.399869 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.399844    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:41:57.533495 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:57.533469    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"]
Jul 29 18:41:57.535688 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:41:57.535658    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409f8a8b_40fa_4720_8f44_ec4a82ea2ee9.slice/crio-66179ddf63393aa6520aca0a0bd46d092b4f5d24d3f0ee7403b0cbd2f7250448 WatchSource:0}: Error finding container 66179ddf63393aa6520aca0a0bd46d092b4f5d24d3f0ee7403b0cbd2f7250448: Status 404 returned error can't find the container with id 66179ddf63393aa6520aca0a0bd46d092b4f5d24d3f0ee7403b0cbd2f7250448
Jul 29 18:41:58.381753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:58.381718    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg" event={"ID":"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9","Type":"ContainerStarted","Data":"66179ddf63393aa6520aca0a0bd46d092b4f5d24d3f0ee7403b0cbd2f7250448"}
Jul 29 18:41:58.454681 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:41:58.454607    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:00.420361 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:00.420324    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:42:00.420623 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:00.420392    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:42:01.398290 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:01.398248    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg" event={"ID":"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9","Type":"ContainerStarted","Data":"631b527af9f68d162a23e446ae0a962714f1fa933453aaab568a8f33752f4d01"}
Jul 29 18:42:01.398555 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:01.398430    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:42:01.400153 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:01.400119    2560 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-g8vpg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 18:42:01.400287 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:01.400181    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg" podUID="409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 18:42:01.418553 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:01.418506    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg" podStartSLOduration=1.5361531419999999 podStartE2EDuration="4.418491398s" totalImagesPullingTime="2.882338256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:01.417655319 +0000 UTC m=+447.311655723" watchObservedRunningTime="2026-07-29 18:42:01.418491398 +0000 UTC m=+447.312491798"
Jul 29 18:42:01.503594 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:01.503558    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:01.618044 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:01.618015    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:02.403271 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:02.403231    2560 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-g8vpg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 18:42:02.403430 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:02.403287    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg" podUID="409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 18:42:02.661286 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:02.661208    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:02.777250 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:02.777218    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:05.404218 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.404166    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:42:05.773078 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.773002    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"]
Jul 29 18:42:05.776083 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.776059    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.780421 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.780399    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-dxs88\""
Jul 29 18:42:05.796742 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.796695    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"]
Jul 29 18:42:05.811796 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.811739    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.811796 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.811802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-credential-socket\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.812046 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.811826    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-workload-socket\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.812046 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.811883    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcthw\" (UniqueName: \"kubernetes.io/projected/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-kube-api-access-wcthw\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.812046 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.811926    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istio-token\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.812046 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.811951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-workload-certs\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.812046 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.812033    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.812231 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.812075    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istio-data\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.812231 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.812093    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.845399 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.845366    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:05.912843 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.912806    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcthw\" (UniqueName: \"kubernetes.io/projected/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-kube-api-access-wcthw\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.912843 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.912848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istio-token\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.913402 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.912876    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-workload-certs\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.913402 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.912903    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.913402 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.912920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istio-data\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.913402 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.912948    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.913402 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.912989    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.913402 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.913034    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-credential-socket\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.913402 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.913064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-workload-socket\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.913402 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.913350    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-workload-certs\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.913750 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.913474    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istio-data\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.913750 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.913514    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-workload-socket\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.913750 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.913612    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-credential-socket\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.913858 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.913773    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.915209 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.915170    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.915500 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.915478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.924084 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.924063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcthw\" (UniqueName: \"kubernetes.io/projected/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-kube-api-access-wcthw\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.924941 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.924924    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2-istio-token\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-w95c8\" (UID: \"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:05.925838 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:05.925817    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:06.088434 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:06.088399    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:06.216908 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:06.216858    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"]
Jul 29 18:42:06.219538 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:42:06.219510    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e400cb_9af3_4b4e_aa6f_6b34561ee5f2.slice/crio-0c3d0a5e4880ef6165db2cade99ecb5c2c4827bc4dc8fd871c4193502f9c84f0 WatchSource:0}: Error finding container 0c3d0a5e4880ef6165db2cade99ecb5c2c4827bc4dc8fd871c4193502f9c84f0: Status 404 returned error can't find the container with id 0c3d0a5e4880ef6165db2cade99ecb5c2c4827bc4dc8fd871c4193502f9c84f0
Jul 29 18:42:06.419337 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:06.419257    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8" event={"ID":"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2","Type":"ContainerStarted","Data":"0c3d0a5e4880ef6165db2cade99ecb5c2c4827bc4dc8fd871c4193502f9c84f0"}
Jul 29 18:42:08.899228 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:08.899176    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:42:08.899527 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:08.899265    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:42:08.899527 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:08.899297    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:42:09.058751 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:09.058716    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:09.433162 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:09.433126    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8" event={"ID":"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2","Type":"ContainerStarted","Data":"ef3219a439a723f51f2d898c0ef8cb0d26691e32ed6cea05c0434d84c64978be"}
Jul 29 18:42:09.454956 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:09.454907    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8" podStartSLOduration=1.7774187380000002 podStartE2EDuration="4.454892324s" totalImagesPullingTime="2.677473586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:09.4544143 +0000 UTC m=+455.348414723" watchObservedRunningTime="2026-07-29 18:42:09.454892324 +0000 UTC m=+455.348892725"
Jul 29 18:42:10.089240 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:10.089187    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:10.094038 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:10.094012    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:10.437267 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:10.437173    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:10.438276 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:10.438255    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:42:11.291474 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:11.291435    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:11.399614 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:11.399579    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:16.102883 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.102841    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"]
Jul 29 18:42:16.106160 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.106142    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"
Jul 29 18:42:16.108694 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.108664    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-drbqn\""
Jul 29 18:42:16.108898 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.108885    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 18:42:16.109503 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.109489    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 18:42:16.115634 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.115612    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"]
Jul 29 18:42:16.202510 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.202471    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7\" (UID: \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"
Jul 29 18:42:16.202665 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.202528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7\" (UID: \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"
Jul 29 18:42:16.202665 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.202618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmg8s\" (UniqueName: \"kubernetes.io/projected/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-kube-api-access-lmg8s\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7\" (UID: \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"
Jul 29 18:42:16.204053 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.204027    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"]
Jul 29 18:42:16.207336 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.207320    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"
Jul 29 18:42:16.217351 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.217328    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"]
Jul 29 18:42:16.303944 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.303913    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7\" (UID: \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"
Jul 29 18:42:16.304127 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.303954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng\" (UID: \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"
Jul 29 18:42:16.304127 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.303978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmg8s\" (UniqueName: \"kubernetes.io/projected/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-kube-api-access-lmg8s\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7\" (UID: \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"
Jul 29 18:42:16.304127 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.304021    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbsqc\" (UniqueName: \"kubernetes.io/projected/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-kube-api-access-kbsqc\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng\" (UID: \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"
Jul 29 18:42:16.304127 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.304059    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7\" (UID: \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"
Jul 29 18:42:16.304127 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.304085    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng\" (UID: \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"
Jul 29 18:42:16.304436 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.304338    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7\" (UID: \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"
Jul 29 18:42:16.304436 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.304418    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7\" (UID: \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"
Jul 29 18:42:16.309185 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.309159    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"]
Jul 29 18:42:16.317508 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.317479    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"
Jul 29 18:42:16.320211 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.320171    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmg8s\" (UniqueName: \"kubernetes.io/projected/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-kube-api-access-lmg8s\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7\" (UID: \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"
Jul 29 18:42:16.323237 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.323214    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"]
Jul 29 18:42:16.403383 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.403305    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"]
Jul 29 18:42:16.404861 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.404836    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0600662-c61e-44d8-baab-18156d3d5daf-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz\" (UID: \"c0600662-c61e-44d8-baab-18156d3d5daf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"
Jul 29 18:42:16.404959 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.404876    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng\" (UID: \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"
Jul 29 18:42:16.404959 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.404939    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0600662-c61e-44d8-baab-18156d3d5daf-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz\" (UID: \"c0600662-c61e-44d8-baab-18156d3d5daf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"
Jul 29 18:42:16.405032 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.404971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsqc\" (UniqueName: \"kubernetes.io/projected/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-kube-api-access-kbsqc\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng\" (UID: \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"
Jul 29 18:42:16.405073 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.405035    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgbpj\" (UniqueName: \"kubernetes.io/projected/c0600662-c61e-44d8-baab-18156d3d5daf-kube-api-access-mgbpj\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz\" (UID: \"c0600662-c61e-44d8-baab-18156d3d5daf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"
Jul 29 18:42:16.405073 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.405065    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng\" (UID: \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"
Jul 29 18:42:16.405169 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.405156    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng\" (UID: \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"
Jul 29 18:42:16.405355 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.405340    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng\" (UID: \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"
Jul 29 18:42:16.406833 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.406819    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"
Jul 29 18:42:16.413840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.413818    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsqc\" (UniqueName: \"kubernetes.io/projected/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-kube-api-access-kbsqc\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng\" (UID: \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"
Jul 29 18:42:16.415763 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.415742    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"
Jul 29 18:42:16.420620 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.420588    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"]
Jul 29 18:42:16.505729 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.505699    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0600662-c61e-44d8-baab-18156d3d5daf-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz\" (UID: \"c0600662-c61e-44d8-baab-18156d3d5daf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"
Jul 29 18:42:16.505882 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.505738    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d7019e-175b-4b51-a2ae-e27e1699e9d6-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5\" (UID: \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"
Jul 29 18:42:16.505882 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.505842    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbpj\" (UniqueName: \"kubernetes.io/projected/c0600662-c61e-44d8-baab-18156d3d5daf-kube-api-access-mgbpj\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz\" (UID: \"c0600662-c61e-44d8-baab-18156d3d5daf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"
Jul 29 18:42:16.505882 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.505868    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t458v\" (UniqueName: \"kubernetes.io/projected/49d7019e-175b-4b51-a2ae-e27e1699e9d6-kube-api-access-t458v\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5\" (UID: \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"
Jul 29 18:42:16.506047 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.506025    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d7019e-175b-4b51-a2ae-e27e1699e9d6-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5\" (UID: \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"
Jul 29 18:42:16.506108 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.506086    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0600662-c61e-44d8-baab-18156d3d5daf-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz\" (UID: \"c0600662-c61e-44d8-baab-18156d3d5daf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"
Jul 29 18:42:16.506161 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.506135    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0600662-c61e-44d8-baab-18156d3d5daf-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz\" (UID: \"c0600662-c61e-44d8-baab-18156d3d5daf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"
Jul 29 18:42:16.506440 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.506421    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0600662-c61e-44d8-baab-18156d3d5daf-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz\" (UID: \"c0600662-c61e-44d8-baab-18156d3d5daf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"
Jul 29 18:42:16.515007 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.514981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbpj\" (UniqueName: \"kubernetes.io/projected/c0600662-c61e-44d8-baab-18156d3d5daf-kube-api-access-mgbpj\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz\" (UID: \"c0600662-c61e-44d8-baab-18156d3d5daf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"
Jul 29 18:42:16.516812 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.516684    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"
Jul 29 18:42:16.547621 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.547587    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"]
Jul 29 18:42:16.549135 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:42:16.549110    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f4793f_b4fc_42ff_a7c7_41ba0404bce6.slice/crio-38bc852ae0d73893cc2343d04d32eec1e89ccec30d9cd69bab9f269c56c85bb0 WatchSource:0}: Error finding container 38bc852ae0d73893cc2343d04d32eec1e89ccec30d9cd69bab9f269c56c85bb0: Status 404 returned error can't find the container with id 38bc852ae0d73893cc2343d04d32eec1e89ccec30d9cd69bab9f269c56c85bb0
Jul 29 18:42:16.606800 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.606768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d7019e-175b-4b51-a2ae-e27e1699e9d6-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5\" (UID: \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"
Jul 29 18:42:16.606971 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.606840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t458v\" (UniqueName: \"kubernetes.io/projected/49d7019e-175b-4b51-a2ae-e27e1699e9d6-kube-api-access-t458v\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5\" (UID: \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"
Jul 29 18:42:16.606971 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.606907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d7019e-175b-4b51-a2ae-e27e1699e9d6-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5\" (UID: \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"
Jul 29 18:42:16.607268 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.607240    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d7019e-175b-4b51-a2ae-e27e1699e9d6-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5\" (UID: \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"
Jul 29 18:42:16.607453 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.607432    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d7019e-175b-4b51-a2ae-e27e1699e9d6-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5\" (UID: \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"
Jul 29 18:42:16.616733 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.616700    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t458v\" (UniqueName: \"kubernetes.io/projected/49d7019e-175b-4b51-a2ae-e27e1699e9d6-kube-api-access-t458v\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5\" (UID: \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"
Jul 29 18:42:16.633286 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.633252    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"
Jul 29 18:42:16.653734 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.653668    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"]
Jul 29 18:42:16.655397 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:42:16.655362    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f740e6_1beb_4def_b8c0_40fb06c7cb5b.slice/crio-325e951727f7f4e08ca1850dba8128245ee7e3eb748477c1d1f2f4faab174411 WatchSource:0}: Error finding container 325e951727f7f4e08ca1850dba8128245ee7e3eb748477c1d1f2f4faab174411: Status 404 returned error can't find the container with id 325e951727f7f4e08ca1850dba8128245ee7e3eb748477c1d1f2f4faab174411
Jul 29 18:42:16.722379 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.722347    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"
Jul 29 18:42:16.768417 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.768228    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"]
Jul 29 18:42:16.773544 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:42:16.773510    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0600662_c61e_44d8_baab_18156d3d5daf.slice/crio-b91fc6a99f4471b558a7db1a0654508fba80572c00c0e2f06d287df0e5cdd60c WatchSource:0}: Error finding container b91fc6a99f4471b558a7db1a0654508fba80572c00c0e2f06d287df0e5cdd60c: Status 404 returned error can't find the container with id b91fc6a99f4471b558a7db1a0654508fba80572c00c0e2f06d287df0e5cdd60c
Jul 29 18:42:16.866265 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:16.866242    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"]
Jul 29 18:42:16.868401 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:42:16.868379    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d7019e_175b_4b51_a2ae_e27e1699e9d6.slice/crio-41604080471a0461b1deefc8fc3ddea333262c09f496bd62d45fcc200865dd32 WatchSource:0}: Error finding container 41604080471a0461b1deefc8fc3ddea333262c09f496bd62d45fcc200865dd32: Status 404 returned error can't find the container with id 41604080471a0461b1deefc8fc3ddea333262c09f496bd62d45fcc200865dd32
Jul 29 18:42:17.466600 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:17.466561    2560 generic.go:356] "Generic (PLEG): container finished" podID="b1f740e6-1beb-4def-b8c0-40fb06c7cb5b" containerID="9f1e294c685d80ab8625368d5c6e13cdeb370202a013b8531329d0d8692a738f" exitCode=0
Jul 29 18:42:17.467010 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:17.466653    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng" event={"ID":"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b","Type":"ContainerDied","Data":"9f1e294c685d80ab8625368d5c6e13cdeb370202a013b8531329d0d8692a738f"}
Jul 29 18:42:17.467010 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:17.466715    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng" event={"ID":"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b","Type":"ContainerStarted","Data":"325e951727f7f4e08ca1850dba8128245ee7e3eb748477c1d1f2f4faab174411"}
Jul 29 18:42:17.468292 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:17.468270    2560 generic.go:356] "Generic (PLEG): container finished" podID="49d7019e-175b-4b51-a2ae-e27e1699e9d6" containerID="f1b022ea4fd59c5ecca97b69e31464eb3aa138055e488f57884c03a7f902db88" exitCode=0
Jul 29 18:42:17.468367 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:17.468345    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5" event={"ID":"49d7019e-175b-4b51-a2ae-e27e1699e9d6","Type":"ContainerDied","Data":"f1b022ea4fd59c5ecca97b69e31464eb3aa138055e488f57884c03a7f902db88"}
Jul 29 18:42:17.468423 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:17.468392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5" event={"ID":"49d7019e-175b-4b51-a2ae-e27e1699e9d6","Type":"ContainerStarted","Data":"41604080471a0461b1deefc8fc3ddea333262c09f496bd62d45fcc200865dd32"}
Jul 29 18:42:17.469914 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:17.469892    2560 generic.go:356] "Generic (PLEG): container finished" podID="c0600662-c61e-44d8-baab-18156d3d5daf" containerID="e1b06214e53a2696cf57d7f2d760ba033b469947fd482620bbb707db02805ce7" exitCode=0
Jul 29 18:42:17.469996 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:17.469974    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz" event={"ID":"c0600662-c61e-44d8-baab-18156d3d5daf","Type":"ContainerDied","Data":"e1b06214e53a2696cf57d7f2d760ba033b469947fd482620bbb707db02805ce7"}
Jul 29 18:42:17.470058 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:17.470036    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz" event={"ID":"c0600662-c61e-44d8-baab-18156d3d5daf","Type":"ContainerStarted","Data":"b91fc6a99f4471b558a7db1a0654508fba80572c00c0e2f06d287df0e5cdd60c"}
Jul 29 18:42:17.471654 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:17.471634    2560 generic.go:356] "Generic (PLEG): container finished" podID="75f4793f-b4fc-42ff-a7c7-41ba0404bce6" containerID="53a3fc6c6ebf2c7453d7ee92515469985500e1444adf9c24970460695f06dffa" exitCode=0
Jul 29 18:42:17.471721 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:17.471701    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7" event={"ID":"75f4793f-b4fc-42ff-a7c7-41ba0404bce6","Type":"ContainerDied","Data":"53a3fc6c6ebf2c7453d7ee92515469985500e1444adf9c24970460695f06dffa"}
Jul 29 18:42:17.471757 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:17.471741    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7" event={"ID":"75f4793f-b4fc-42ff-a7c7-41ba0404bce6","Type":"ContainerStarted","Data":"38bc852ae0d73893cc2343d04d32eec1e89ccec30d9cd69bab9f269c56c85bb0"}
Jul 29 18:42:19.482570 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:19.482536    2560 generic.go:356] "Generic (PLEG): container finished" podID="b1f740e6-1beb-4def-b8c0-40fb06c7cb5b" containerID="531d8fa9f6c867563b5f156c05fa1b6b5f8ef5be08fb6940661c886f81241046" exitCode=0
Jul 29 18:42:19.483005 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:19.482619    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng" event={"ID":"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b","Type":"ContainerDied","Data":"531d8fa9f6c867563b5f156c05fa1b6b5f8ef5be08fb6940661c886f81241046"}
Jul 29 18:42:19.484435 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:19.484414    2560 generic.go:356] "Generic (PLEG): container finished" podID="49d7019e-175b-4b51-a2ae-e27e1699e9d6" containerID="5a922399fb7f556b3717366fd0f2df66f3562e24d310f35eb72f81fa98859a3e" exitCode=0
Jul 29 18:42:19.484533 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:19.484495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5" event={"ID":"49d7019e-175b-4b51-a2ae-e27e1699e9d6","Type":"ContainerDied","Data":"5a922399fb7f556b3717366fd0f2df66f3562e24d310f35eb72f81fa98859a3e"}
Jul 29 18:42:19.486417 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:19.486396    2560 generic.go:356] "Generic (PLEG): container finished" podID="c0600662-c61e-44d8-baab-18156d3d5daf" containerID="e2f302429aabb97db1a66d41e973af51a8a94d0e86bfba117780e28f35674124" exitCode=0
Jul 29 18:42:19.486514 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:19.486482    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz" event={"ID":"c0600662-c61e-44d8-baab-18156d3d5daf","Type":"ContainerDied","Data":"e2f302429aabb97db1a66d41e973af51a8a94d0e86bfba117780e28f35674124"}
Jul 29 18:42:19.493663 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:19.493643    2560 generic.go:356] "Generic (PLEG): container finished" podID="75f4793f-b4fc-42ff-a7c7-41ba0404bce6" containerID="53e7bedded8c28afcf8f41fc57581f5ecfdac3c431e512ac3621435b8b24816f" exitCode=0
Jul 29 18:42:19.493758 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:19.493699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7" event={"ID":"75f4793f-b4fc-42ff-a7c7-41ba0404bce6","Type":"ContainerDied","Data":"53e7bedded8c28afcf8f41fc57581f5ecfdac3c431e512ac3621435b8b24816f"}
Jul 29 18:42:20.500486 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:20.500453    2560 generic.go:356] "Generic (PLEG): container finished" podID="b1f740e6-1beb-4def-b8c0-40fb06c7cb5b" containerID="aa3169756430366465d0db25fbd1088365c1494b2078d77c5f3fdb552128ad8d" exitCode=0
Jul 29 18:42:20.500865 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:20.500535    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng" event={"ID":"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b","Type":"ContainerDied","Data":"aa3169756430366465d0db25fbd1088365c1494b2078d77c5f3fdb552128ad8d"}
Jul 29 18:42:20.502371 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:20.502345    2560 generic.go:356] "Generic (PLEG): container finished" podID="49d7019e-175b-4b51-a2ae-e27e1699e9d6" containerID="0cc0a543128545386d29e379900c7514c889b398230923e490e87670865cd0d3" exitCode=0
Jul 29 18:42:20.502500 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:20.502395    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5" event={"ID":"49d7019e-175b-4b51-a2ae-e27e1699e9d6","Type":"ContainerDied","Data":"0cc0a543128545386d29e379900c7514c889b398230923e490e87670865cd0d3"}
Jul 29 18:42:20.504311 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:20.504291    2560 generic.go:356] "Generic (PLEG): container finished" podID="c0600662-c61e-44d8-baab-18156d3d5daf" containerID="bedb7129b50a9a17be862c6d00897fb71c63405b7e259832a0b0787de5e7c115" exitCode=0
Jul 29 18:42:20.504391 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:20.504317    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz" event={"ID":"c0600662-c61e-44d8-baab-18156d3d5daf","Type":"ContainerDied","Data":"bedb7129b50a9a17be862c6d00897fb71c63405b7e259832a0b0787de5e7c115"}
Jul 29 18:42:20.506260 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:20.506239    2560 generic.go:356] "Generic (PLEG): container finished" podID="75f4793f-b4fc-42ff-a7c7-41ba0404bce6" containerID="f18b89b702e99cfba08fca328c710e3b3e5295812737445f1ff5c2b162a89303" exitCode=0
Jul 29 18:42:20.506336 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:20.506279    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7" event={"ID":"75f4793f-b4fc-42ff-a7c7-41ba0404bce6","Type":"ContainerDied","Data":"f18b89b702e99cfba08fca328c710e3b3e5295812737445f1ff5c2b162a89303"}
Jul 29 18:42:21.635630 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.635606    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"
Jul 29 18:42:21.675347 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.675324    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"
Jul 29 18:42:21.713929 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.713905    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"
Jul 29 18:42:21.716369 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.716343    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"
Jul 29 18:42:21.748444 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.748417    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t458v\" (UniqueName: \"kubernetes.io/projected/49d7019e-175b-4b51-a2ae-e27e1699e9d6-kube-api-access-t458v\") pod \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\" (UID: \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\") "
Jul 29 18:42:21.748588 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.748455    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-util\") pod \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\" (UID: \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\") "
Jul 29 18:42:21.748588 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.748489    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmg8s\" (UniqueName: \"kubernetes.io/projected/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-kube-api-access-lmg8s\") pod \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\" (UID: \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\") "
Jul 29 18:42:21.748718 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.748601    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d7019e-175b-4b51-a2ae-e27e1699e9d6-bundle\") pod \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\" (UID: \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\") "
Jul 29 18:42:21.748718 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.748654    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-bundle\") pod \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\" (UID: \"75f4793f-b4fc-42ff-a7c7-41ba0404bce6\") "
Jul 29 18:42:21.748718 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.748712    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d7019e-175b-4b51-a2ae-e27e1699e9d6-util\") pod \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\" (UID: \"49d7019e-175b-4b51-a2ae-e27e1699e9d6\") "
Jul 29 18:42:21.749064 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.749036    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d7019e-175b-4b51-a2ae-e27e1699e9d6-bundle" (OuterVolumeSpecName: "bundle") pod "49d7019e-175b-4b51-a2ae-e27e1699e9d6" (UID: "49d7019e-175b-4b51-a2ae-e27e1699e9d6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:42:21.749621 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.749589    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-bundle" (OuterVolumeSpecName: "bundle") pod "75f4793f-b4fc-42ff-a7c7-41ba0404bce6" (UID: "75f4793f-b4fc-42ff-a7c7-41ba0404bce6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:42:21.750684 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.750666    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d7019e-175b-4b51-a2ae-e27e1699e9d6-kube-api-access-t458v" (OuterVolumeSpecName: "kube-api-access-t458v") pod "49d7019e-175b-4b51-a2ae-e27e1699e9d6" (UID: "49d7019e-175b-4b51-a2ae-e27e1699e9d6"). InnerVolumeSpecName "kube-api-access-t458v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:42:21.750872 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.750856    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-kube-api-access-lmg8s" (OuterVolumeSpecName: "kube-api-access-lmg8s") pod "75f4793f-b4fc-42ff-a7c7-41ba0404bce6" (UID: "75f4793f-b4fc-42ff-a7c7-41ba0404bce6"). InnerVolumeSpecName "kube-api-access-lmg8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:42:21.755280 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.755173    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-util" (OuterVolumeSpecName: "util") pod "75f4793f-b4fc-42ff-a7c7-41ba0404bce6" (UID: "75f4793f-b4fc-42ff-a7c7-41ba0404bce6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:42:21.756680 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.756634    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d7019e-175b-4b51-a2ae-e27e1699e9d6-util" (OuterVolumeSpecName: "util") pod "49d7019e-175b-4b51-a2ae-e27e1699e9d6" (UID: "49d7019e-175b-4b51-a2ae-e27e1699e9d6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:42:21.849394 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.849361    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbsqc\" (UniqueName: \"kubernetes.io/projected/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-kube-api-access-kbsqc\") pod \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\" (UID: \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\") "
Jul 29 18:42:21.849575 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.849457    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0600662-c61e-44d8-baab-18156d3d5daf-util\") pod \"c0600662-c61e-44d8-baab-18156d3d5daf\" (UID: \"c0600662-c61e-44d8-baab-18156d3d5daf\") "
Jul 29 18:42:21.849575 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.849481    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgbpj\" (UniqueName: \"kubernetes.io/projected/c0600662-c61e-44d8-baab-18156d3d5daf-kube-api-access-mgbpj\") pod \"c0600662-c61e-44d8-baab-18156d3d5daf\" (UID: \"c0600662-c61e-44d8-baab-18156d3d5daf\") "
Jul 29 18:42:21.849575 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.849514    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0600662-c61e-44d8-baab-18156d3d5daf-bundle\") pod \"c0600662-c61e-44d8-baab-18156d3d5daf\" (UID: \"c0600662-c61e-44d8-baab-18156d3d5daf\") "
Jul 29 18:42:21.849575 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.849548    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-util\") pod \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\" (UID: \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\") "
Jul 29 18:42:21.849793 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.849576    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-bundle\") pod \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\" (UID: \"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b\") "
Jul 29 18:42:21.849845 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.849815    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmg8s\" (UniqueName: \"kubernetes.io/projected/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-kube-api-access-lmg8s\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:42:21.849845 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.849833    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d7019e-175b-4b51-a2ae-e27e1699e9d6-bundle\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:42:21.849948 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.849844    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-bundle\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:42:21.849948 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.849853    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d7019e-175b-4b51-a2ae-e27e1699e9d6-util\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:42:21.849948 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.849863    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t458v\" (UniqueName: \"kubernetes.io/projected/49d7019e-175b-4b51-a2ae-e27e1699e9d6-kube-api-access-t458v\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:42:21.849948 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.849871    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75f4793f-b4fc-42ff-a7c7-41ba0404bce6-util\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:42:21.850133 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.850099    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-bundle" (OuterVolumeSpecName: "bundle") pod "b1f740e6-1beb-4def-b8c0-40fb06c7cb5b" (UID: "b1f740e6-1beb-4def-b8c0-40fb06c7cb5b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:42:21.850183 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.850106    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0600662-c61e-44d8-baab-18156d3d5daf-bundle" (OuterVolumeSpecName: "bundle") pod "c0600662-c61e-44d8-baab-18156d3d5daf" (UID: "c0600662-c61e-44d8-baab-18156d3d5daf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:42:21.851651 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.851625    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-kube-api-access-kbsqc" (OuterVolumeSpecName: "kube-api-access-kbsqc") pod "b1f740e6-1beb-4def-b8c0-40fb06c7cb5b" (UID: "b1f740e6-1beb-4def-b8c0-40fb06c7cb5b"). InnerVolumeSpecName "kube-api-access-kbsqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:42:21.851754 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.851649    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0600662-c61e-44d8-baab-18156d3d5daf-kube-api-access-mgbpj" (OuterVolumeSpecName: "kube-api-access-mgbpj") pod "c0600662-c61e-44d8-baab-18156d3d5daf" (UID: "c0600662-c61e-44d8-baab-18156d3d5daf"). InnerVolumeSpecName "kube-api-access-mgbpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:42:21.855335 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.855299    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0600662-c61e-44d8-baab-18156d3d5daf-util" (OuterVolumeSpecName: "util") pod "c0600662-c61e-44d8-baab-18156d3d5daf" (UID: "c0600662-c61e-44d8-baab-18156d3d5daf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:42:21.951105 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.951065    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbsqc\" (UniqueName: \"kubernetes.io/projected/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-kube-api-access-kbsqc\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:42:21.951105 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.951095    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0600662-c61e-44d8-baab-18156d3d5daf-util\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:42:21.951105 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.951105    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgbpj\" (UniqueName: \"kubernetes.io/projected/c0600662-c61e-44d8-baab-18156d3d5daf-kube-api-access-mgbpj\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:42:21.951105 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.951115    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0600662-c61e-44d8-baab-18156d3d5daf-bundle\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:42:21.951382 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.951123    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-bundle\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:42:21.979896 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:21.979851    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-util" (OuterVolumeSpecName: "util") pod "b1f740e6-1beb-4def-b8c0-40fb06c7cb5b" (UID: "b1f740e6-1beb-4def-b8c0-40fb06c7cb5b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:42:22.052122 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:22.052089    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1f740e6-1beb-4def-b8c0-40fb06c7cb5b-util\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:42:22.518014 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:22.517973    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5" event={"ID":"49d7019e-175b-4b51-a2ae-e27e1699e9d6","Type":"ContainerDied","Data":"41604080471a0461b1deefc8fc3ddea333262c09f496bd62d45fcc200865dd32"}
Jul 29 18:42:22.518014 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:22.518000    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sfzp5"
Jul 29 18:42:22.518261 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:22.518026    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41604080471a0461b1deefc8fc3ddea333262c09f496bd62d45fcc200865dd32"
Jul 29 18:42:22.519810 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:22.519785    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz" event={"ID":"c0600662-c61e-44d8-baab-18156d3d5daf","Type":"ContainerDied","Data":"b91fc6a99f4471b558a7db1a0654508fba80572c00c0e2f06d287df0e5cdd60c"}
Jul 29 18:42:22.519937 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:22.519830    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b91fc6a99f4471b558a7db1a0654508fba80572c00c0e2f06d287df0e5cdd60c"
Jul 29 18:42:22.519937 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:22.519789    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkgqhz"
Jul 29 18:42:22.528085 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:22.528062    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7"
Jul 29 18:42:22.528209 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:22.528078    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053nk4f7" event={"ID":"75f4793f-b4fc-42ff-a7c7-41ba0404bce6","Type":"ContainerDied","Data":"38bc852ae0d73893cc2343d04d32eec1e89ccec30d9cd69bab9f269c56c85bb0"}
Jul 29 18:42:22.528209 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:22.528151    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38bc852ae0d73893cc2343d04d32eec1e89ccec30d9cd69bab9f269c56c85bb0"
Jul 29 18:42:22.530309 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:22.530290    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng" event={"ID":"b1f740e6-1beb-4def-b8c0-40fb06c7cb5b","Type":"ContainerDied","Data":"325e951727f7f4e08ca1850dba8128245ee7e3eb748477c1d1f2f4faab174411"}
Jul 29 18:42:22.530428 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:22.530328    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="325e951727f7f4e08ca1850dba8128245ee7e3eb748477c1d1f2f4faab174411"
Jul 29 18:42:22.530428 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:22.530329    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cht2ng"
Jul 29 18:42:24.081118 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:24.081082    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:25.680728 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:25.680699    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:25.776962 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:25.776932    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:25.876810 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:25.876778    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:26.514363 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:26.514321    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:26.611963 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:26.611909    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:28.553180 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553143    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-7vlkb"]
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553502    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f4793f-b4fc-42ff-a7c7-41ba0404bce6" containerName="util"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553515    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f4793f-b4fc-42ff-a7c7-41ba0404bce6" containerName="util"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553524    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d7019e-175b-4b51-a2ae-e27e1699e9d6" containerName="pull"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553530    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d7019e-175b-4b51-a2ae-e27e1699e9d6" containerName="pull"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553546    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f4793f-b4fc-42ff-a7c7-41ba0404bce6" containerName="pull"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553551    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f4793f-b4fc-42ff-a7c7-41ba0404bce6" containerName="pull"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553560    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d7019e-175b-4b51-a2ae-e27e1699e9d6" containerName="util"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553565    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d7019e-175b-4b51-a2ae-e27e1699e9d6" containerName="util"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553572    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0600662-c61e-44d8-baab-18156d3d5daf" containerName="pull"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553578    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0600662-c61e-44d8-baab-18156d3d5daf" containerName="pull"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553584    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0600662-c61e-44d8-baab-18156d3d5daf" containerName="util"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553589    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0600662-c61e-44d8-baab-18156d3d5daf" containerName="util"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553598    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1f740e6-1beb-4def-b8c0-40fb06c7cb5b" containerName="pull"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553603    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1f740e6-1beb-4def-b8c0-40fb06c7cb5b" containerName="pull"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553608    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f4793f-b4fc-42ff-a7c7-41ba0404bce6" containerName="extract"
Jul 29 18:42:28.553605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553614    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f4793f-b4fc-42ff-a7c7-41ba0404bce6" containerName="extract"
Jul 29 18:42:28.554045 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553621    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1f740e6-1beb-4def-b8c0-40fb06c7cb5b" containerName="util"
Jul 29 18:42:28.554045 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553626    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1f740e6-1beb-4def-b8c0-40fb06c7cb5b" containerName="util"
Jul 29 18:42:28.554045 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553633    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0600662-c61e-44d8-baab-18156d3d5daf" containerName="extract"
Jul 29 18:42:28.554045 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553638    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0600662-c61e-44d8-baab-18156d3d5daf" containerName="extract"
Jul 29 18:42:28.554045 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553648    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1f740e6-1beb-4def-b8c0-40fb06c7cb5b" containerName="extract"
Jul 29 18:42:28.554045 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553653    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1f740e6-1beb-4def-b8c0-40fb06c7cb5b" containerName="extract"
Jul 29 18:42:28.554045 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553658    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d7019e-175b-4b51-a2ae-e27e1699e9d6" containerName="extract"
Jul 29 18:42:28.554045 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553663    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d7019e-175b-4b51-a2ae-e27e1699e9d6" containerName="extract"
Jul 29 18:42:28.554045 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553713    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f4793f-b4fc-42ff-a7c7-41ba0404bce6" containerName="extract"
Jul 29 18:42:28.554045 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553719    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1f740e6-1beb-4def-b8c0-40fb06c7cb5b" containerName="extract"
Jul 29 18:42:28.554045 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553725    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="49d7019e-175b-4b51-a2ae-e27e1699e9d6" containerName="extract"
Jul 29 18:42:28.554045 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.553731    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0600662-c61e-44d8-baab-18156d3d5daf" containerName="extract"
Jul 29 18:42:28.562944 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.562922    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-7vlkb"
Jul 29 18:42:28.565905 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.565720    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 18:42:28.565905 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.565741    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 18:42:28.565905 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.565746    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-68wxh\""
Jul 29 18:42:28.565905 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.565778    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 29 18:42:28.567044 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.567020    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-7vlkb"]
Jul 29 18:42:28.709871 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.709838    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvxp\" (UniqueName: \"kubernetes.io/projected/a262ea76-3d0a-4db4-8dcd-8c9ef79419f1-kube-api-access-vlvxp\") pod \"dns-operator-controller-manager-ccbdf686-7vlkb\" (UID: \"a262ea76-3d0a-4db4-8dcd-8c9ef79419f1\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-7vlkb"
Jul 29 18:42:28.810579 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.810550    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvxp\" (UniqueName: \"kubernetes.io/projected/a262ea76-3d0a-4db4-8dcd-8c9ef79419f1-kube-api-access-vlvxp\") pod \"dns-operator-controller-manager-ccbdf686-7vlkb\" (UID: \"a262ea76-3d0a-4db4-8dcd-8c9ef79419f1\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-7vlkb"
Jul 29 18:42:28.821685 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.821646    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvxp\" (UniqueName: \"kubernetes.io/projected/a262ea76-3d0a-4db4-8dcd-8c9ef79419f1-kube-api-access-vlvxp\") pod \"dns-operator-controller-manager-ccbdf686-7vlkb\" (UID: \"a262ea76-3d0a-4db4-8dcd-8c9ef79419f1\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-7vlkb"
Jul 29 18:42:28.873534 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:28.873509    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-7vlkb"
Jul 29 18:42:29.032505 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:29.032385    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-7vlkb"]
Jul 29 18:42:29.035280 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:42:29.035253    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda262ea76_3d0a_4db4_8dcd_8c9ef79419f1.slice/crio-84b7f89d92c2ebc5d91235092168955eeaee91a9ccf38cfd33cb5646a876c2ad WatchSource:0}: Error finding container 84b7f89d92c2ebc5d91235092168955eeaee91a9ccf38cfd33cb5646a876c2ad: Status 404 returned error can't find the container with id 84b7f89d92c2ebc5d91235092168955eeaee91a9ccf38cfd33cb5646a876c2ad
Jul 29 18:42:29.373423 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:29.373385    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:29.564644 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:29.564610    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-7vlkb" event={"ID":"a262ea76-3d0a-4db4-8dcd-8c9ef79419f1","Type":"ContainerStarted","Data":"84b7f89d92c2ebc5d91235092168955eeaee91a9ccf38cfd33cb5646a876c2ad"}
Jul 29 18:42:30.559994 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:30.559961    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-8gnc8"]
Jul 29 18:42:30.565044 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:30.565016    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-8gnc8"
Jul 29 18:42:30.567593 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:30.567570    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-l57gs\""
Jul 29 18:42:30.573419 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:30.573389    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-8gnc8"]
Jul 29 18:42:30.625854 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:30.625813    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8j75\" (UniqueName: \"kubernetes.io/projected/877d5c15-68fa-4dad-b51f-ad6c5f154f58-kube-api-access-x8j75\") pod \"limitador-operator-controller-manager-774d94476f-8gnc8\" (UID: \"877d5c15-68fa-4dad-b51f-ad6c5f154f58\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-8gnc8"
Jul 29 18:42:30.726574 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:30.726533    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8j75\" (UniqueName: \"kubernetes.io/projected/877d5c15-68fa-4dad-b51f-ad6c5f154f58-kube-api-access-x8j75\") pod \"limitador-operator-controller-manager-774d94476f-8gnc8\" (UID: \"877d5c15-68fa-4dad-b51f-ad6c5f154f58\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-8gnc8"
Jul 29 18:42:30.738706 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:30.738680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8j75\" (UniqueName: \"kubernetes.io/projected/877d5c15-68fa-4dad-b51f-ad6c5f154f58-kube-api-access-x8j75\") pod \"limitador-operator-controller-manager-774d94476f-8gnc8\" (UID: \"877d5c15-68fa-4dad-b51f-ad6c5f154f58\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-8gnc8"
Jul 29 18:42:30.878452 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:30.878378    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-8gnc8"
Jul 29 18:42:31.088439 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:31.088413    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:31.247053 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:31.247027    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-8gnc8"]
Jul 29 18:42:31.248875 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:42:31.248844    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877d5c15_68fa_4dad_b51f_ad6c5f154f58.slice/crio-149d918eb5b1a9925a7eb02bb9de3ae6faedf34f0e53c964694a5d4a63fdb638 WatchSource:0}: Error finding container 149d918eb5b1a9925a7eb02bb9de3ae6faedf34f0e53c964694a5d4a63fdb638: Status 404 returned error can't find the container with id 149d918eb5b1a9925a7eb02bb9de3ae6faedf34f0e53c964694a5d4a63fdb638
Jul 29 18:42:31.341236 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:31.341181    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:31.493610 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:31.493540    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:31.579215 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:31.579158    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-8gnc8" event={"ID":"877d5c15-68fa-4dad-b51f-ad6c5f154f58","Type":"ContainerStarted","Data":"149d918eb5b1a9925a7eb02bb9de3ae6faedf34f0e53c964694a5d4a63fdb638"}
Jul 29 18:42:32.091100 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:32.090952    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:32.585395 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:32.585356    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-7vlkb" event={"ID":"a262ea76-3d0a-4db4-8dcd-8c9ef79419f1","Type":"ContainerStarted","Data":"b8445eb75743922f220917d0c8f04d00a023d049faabfb4c91c464e873e14604"}
Jul 29 18:42:32.585920 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:32.585492    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-7vlkb"
Jul 29 18:42:32.588105 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:32.588086    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:32.606356 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:32.606309    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-7vlkb" podStartSLOduration=1.826463207 podStartE2EDuration="4.606296446s" totalImagesPullingTime="2.779833239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:32.604486556 +0000 UTC m=+478.498486995" watchObservedRunningTime="2026-07-29 18:42:32.606296446 +0000 UTC m=+478.500296851"
Jul 29 18:42:33.566174 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:33.566148    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:34.598222 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:34.598163    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-8gnc8" event={"ID":"877d5c15-68fa-4dad-b51f-ad6c5f154f58","Type":"ContainerStarted","Data":"bf5c5f34dad33011a80235d06ef83795307d81b5a8588f80e916c7c716ad8d2d"}
Jul 29 18:42:34.598647 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:34.598276    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-8gnc8"
Jul 29 18:42:34.618521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:34.618466    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-8gnc8" podStartSLOduration=2.331764828 podStartE2EDuration="4.618450263s" totalImagesPullingTime="2.286685435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:34.615754728 +0000 UTC m=+480.509755131" watchObservedRunningTime="2026-07-29 18:42:34.618450263 +0000 UTC m=+480.512450734"
Jul 29 18:42:35.310321 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:35.310285    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:35.703519 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:35.703427    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:36.539963 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:36.539931    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:36.639178 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:36.639151    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:36.742862 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:36.742828    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:37.574516 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:37.574464    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:37.678356 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:37.678322    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-gdd8w"]
Jul 29 18:42:37.680657 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:37.680640    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-gdd8w"
Jul 29 18:42:37.683558 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:37.683531    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-p95h6\""
Jul 29 18:42:37.699434 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:37.699404    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-gdd8w"]
Jul 29 18:42:37.786499 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:37.786463    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqnsw\" (UniqueName: \"kubernetes.io/projected/affc3e91-e990-48f0-93f8-361c8072a238-kube-api-access-qqnsw\") pod \"authorino-operator-56fb6f6856-gdd8w\" (UID: \"affc3e91-e990-48f0-93f8-361c8072a238\") " pod="kuadrant-system/authorino-operator-56fb6f6856-gdd8w"
Jul 29 18:42:37.887407 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:37.887324    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqnsw\" (UniqueName: \"kubernetes.io/projected/affc3e91-e990-48f0-93f8-361c8072a238-kube-api-access-qqnsw\") pod \"authorino-operator-56fb6f6856-gdd8w\" (UID: \"affc3e91-e990-48f0-93f8-361c8072a238\") " pod="kuadrant-system/authorino-operator-56fb6f6856-gdd8w"
Jul 29 18:42:37.902524 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:37.902489    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqnsw\" (UniqueName: \"kubernetes.io/projected/affc3e91-e990-48f0-93f8-361c8072a238-kube-api-access-qqnsw\") pod \"authorino-operator-56fb6f6856-gdd8w\" (UID: \"affc3e91-e990-48f0-93f8-361c8072a238\") " pod="kuadrant-system/authorino-operator-56fb6f6856-gdd8w"
Jul 29 18:42:37.990964 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:37.990934    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-gdd8w"
Jul 29 18:42:38.120761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:38.120735    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-gdd8w"]
Jul 29 18:42:38.123232 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:42:38.123188    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaffc3e91_e990_48f0_93f8_361c8072a238.slice/crio-f6bb3bf2cd9298c8ffa2124bec31b6a33c84f2b32058a9426710a341ecae372a WatchSource:0}: Error finding container f6bb3bf2cd9298c8ffa2124bec31b6a33c84f2b32058a9426710a341ecae372a: Status 404 returned error can't find the container with id f6bb3bf2cd9298c8ffa2124bec31b6a33c84f2b32058a9426710a341ecae372a
Jul 29 18:42:38.619533 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:38.619497    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-gdd8w" event={"ID":"affc3e91-e990-48f0-93f8-361c8072a238","Type":"ContainerStarted","Data":"f6bb3bf2cd9298c8ffa2124bec31b6a33c84f2b32058a9426710a341ecae372a"}
Jul 29 18:42:39.108437 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:39.108408    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:39.208994 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:39.208960    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:39.625962 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:39.625928    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-gdd8w" event={"ID":"affc3e91-e990-48f0-93f8-361c8072a238","Type":"ContainerStarted","Data":"cb3a5aee6804f8b5c6a05654e0dfd303522f1b534c631f2a41525dececefddf7"}
Jul 29 18:42:39.626136 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:39.626058    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-gdd8w"
Jul 29 18:42:39.646969 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:39.646877    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-56fb6f6856-gdd8w" podStartSLOduration=1.381328914 podStartE2EDuration="2.64685968s" totalImagesPullingTime="1.265530766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:39.645063369 +0000 UTC m=+485.539063786" watchObservedRunningTime="2026-07-29 18:42:39.64685968 +0000 UTC m=+485.540860080"
Jul 29 18:42:42.281626 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:42.281589    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:42.502299 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:42.502258    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:42.904431 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:42.904388    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:43.007409 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:43.007362    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:43.594998 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:43.594968    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-7vlkb"
Jul 29 18:42:45.606343 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:45.606312    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-8gnc8"
Jul 29 18:42:46.714305 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:46.714270    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:46.811935 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:46.811904    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:46.912081 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:46.912049    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:47.013584 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:47.013478    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:50.633065 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:50.633037    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-gdd8w"
Jul 29 18:42:55.535290 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:55.535252    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:55.635952 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:55.635917    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:58.138110 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:58.138064    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:42:58.239659 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:42:58.239626    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:01.333606 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:01.333575    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:01.436087 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:01.436048    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:03.939007 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:03.938980    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:04.037903 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:04.037869    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:05.798604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:05.798572    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:05.944349 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:05.944302    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:06.045653 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:06.045617    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:06.149533 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:06.149496    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:06.239426 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:06.239393    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:10.209871 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:10.209837    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:10.270347 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:10.270314    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:14.579340 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:14.579306    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:14.679488 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:14.679458    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:23.900365 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:23.900330    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:24.014160 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.014102    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:24.072849 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.072816    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:24.279217 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.279120    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-47frn"]
Jul 29 18:43:24.283585 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.283564    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-47frn"
Jul 29 18:43:24.286190 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.286168    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2dlbv\""
Jul 29 18:43:24.286317 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.286215    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 29 18:43:24.292548 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.292527    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-47frn"]
Jul 29 18:43:24.323361 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.323333    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-47frn"]
Jul 29 18:43:24.350528 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.350497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scs5l\" (UniqueName: \"kubernetes.io/projected/1f129975-2ae4-4482-9ee0-fb04916afecb-kube-api-access-scs5l\") pod \"limitador-limitador-85d9db4d5d-47frn\" (UID: \"1f129975-2ae4-4482-9ee0-fb04916afecb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-47frn"
Jul 29 18:43:24.350652 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.350546    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f129975-2ae4-4482-9ee0-fb04916afecb-config-file\") pod \"limitador-limitador-85d9db4d5d-47frn\" (UID: \"1f129975-2ae4-4482-9ee0-fb04916afecb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-47frn"
Jul 29 18:43:24.451580 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.451534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f129975-2ae4-4482-9ee0-fb04916afecb-config-file\") pod \"limitador-limitador-85d9db4d5d-47frn\" (UID: \"1f129975-2ae4-4482-9ee0-fb04916afecb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-47frn"
Jul 29 18:43:24.451799 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.451640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scs5l\" (UniqueName: \"kubernetes.io/projected/1f129975-2ae4-4482-9ee0-fb04916afecb-kube-api-access-scs5l\") pod \"limitador-limitador-85d9db4d5d-47frn\" (UID: \"1f129975-2ae4-4482-9ee0-fb04916afecb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-47frn"
Jul 29 18:43:24.452423 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.452395    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f129975-2ae4-4482-9ee0-fb04916afecb-config-file\") pod \"limitador-limitador-85d9db4d5d-47frn\" (UID: \"1f129975-2ae4-4482-9ee0-fb04916afecb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-47frn"
Jul 29 18:43:24.459662 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.459637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scs5l\" (UniqueName: \"kubernetes.io/projected/1f129975-2ae4-4482-9ee0-fb04916afecb-kube-api-access-scs5l\") pod \"limitador-limitador-85d9db4d5d-47frn\" (UID: \"1f129975-2ae4-4482-9ee0-fb04916afecb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-47frn"
Jul 29 18:43:24.594595 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.594557    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-47frn"
Jul 29 18:43:24.728114 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.728085    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-47frn"]
Jul 29 18:43:24.730082 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:43:24.730057    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f129975_2ae4_4482_9ee0_fb04916afecb.slice/crio-c597b0f7c49b197f3eb764ce35171ab44ec7859276d45b332381bc68406e0e92 WatchSource:0}: Error finding container c597b0f7c49b197f3eb764ce35171ab44ec7859276d45b332381bc68406e0e92: Status 404 returned error can't find the container with id c597b0f7c49b197f3eb764ce35171ab44ec7859276d45b332381bc68406e0e92
Jul 29 18:43:24.742290 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.742270    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:24.820275 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.820220    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:24.829513 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:24.829484    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-47frn" event={"ID":"1f129975-2ae4-4482-9ee0-fb04916afecb","Type":"ContainerStarted","Data":"c597b0f7c49b197f3eb764ce35171ab44ec7859276d45b332381bc68406e0e92"}
Jul 29 18:43:26.841603 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:26.841569    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-47frn" event={"ID":"1f129975-2ae4-4482-9ee0-fb04916afecb","Type":"ContainerStarted","Data":"2236814fef8df10e8b3f2926a980b85648cc296b6c46389d2cb65884db99c994"}
Jul 29 18:43:26.842073 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:26.841728    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-47frn"
Jul 29 18:43:26.858524 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:26.858480    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-47frn" podStartSLOduration=1.209876201 podStartE2EDuration="2.85846556s" totalImagesPullingTime="1.648589359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:43:26.858421275 +0000 UTC m=+532.752421675" watchObservedRunningTime="2026-07-29 18:43:26.85846556 +0000 UTC m=+532.752465960"
Jul 29 18:43:33.445094 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:33.445056    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:33.504170 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:33.504136    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:37.848501 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:37.848469    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-47frn"
Jul 29 18:43:44.238346 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:44.238309    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:44.742970 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:44.742935    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:52.804967 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:52.804922    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:43:52.908309 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:43:52.908274    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:44:01.534338 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:01.534298    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:44:06.824786 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:06.824747    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"]
Jul 29 18:44:06.825242 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:06.824970    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg" podUID="409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" containerName="discovery" containerID="cri-o://631b527af9f68d162a23e446ae0a962714f1fa933453aaab568a8f33752f4d01" gracePeriod=30
Jul 29 18:44:07.032687 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.032657    2560 generic.go:356] "Generic (PLEG): container finished" podID="409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" containerID="631b527af9f68d162a23e446ae0a962714f1fa933453aaab568a8f33752f4d01" exitCode=0
Jul 29 18:44:07.032851 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.032736    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg" event={"ID":"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9","Type":"ContainerDied","Data":"631b527af9f68d162a23e446ae0a962714f1fa933453aaab568a8f33752f4d01"}
Jul 29 18:44:07.075597 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.075545    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:44:07.106329 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.106299    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-kubeconfig\") pod \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") "
Jul 29 18:44:07.106476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.106341    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-979w8\" (UniqueName: \"kubernetes.io/projected/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-kube-api-access-979w8\") pod \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") "
Jul 29 18:44:07.106476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.106379    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-csr-ca-configmap\") pod \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") "
Jul 29 18:44:07.106476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.106417    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-local-certs\") pod \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") "
Jul 29 18:44:07.106476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.106442    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-csr-dns-cert\") pod \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") "
Jul 29 18:44:07.106476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.106473    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-token\") pod \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") "
Jul 29 18:44:07.106736 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.106516    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-cacerts\") pod \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\" (UID: \"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9\") "
Jul 29 18:44:07.106869 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.106840    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" (UID: "409f8a8b-40fa-4720-8f44-ec4a82ea2ee9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 18:44:07.109052 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.109018    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-kube-api-access-979w8" (OuterVolumeSpecName: "kube-api-access-979w8") pod "409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" (UID: "409f8a8b-40fa-4720-8f44-ec4a82ea2ee9"). InnerVolumeSpecName "kube-api-access-979w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:44:07.109186 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.109050    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" (UID: "409f8a8b-40fa-4720-8f44-ec4a82ea2ee9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:44:07.109186 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.109066    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-token" (OuterVolumeSpecName: "istio-token") pod "409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" (UID: "409f8a8b-40fa-4720-8f44-ec4a82ea2ee9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:44:07.109186 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.109088    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-local-certs" (OuterVolumeSpecName: "local-certs") pod "409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" (UID: "409f8a8b-40fa-4720-8f44-ec4a82ea2ee9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:44:07.109186 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.109087    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-cacerts" (OuterVolumeSpecName: "cacerts") pod "409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" (UID: "409f8a8b-40fa-4720-8f44-ec4a82ea2ee9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:44:07.109351 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.109225    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" (UID: "409f8a8b-40fa-4720-8f44-ec4a82ea2ee9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:44:07.207674 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.207637    2560 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-local-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:44:07.207674 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.207666    2560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-csr-dns-cert\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:44:07.207674 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.207677    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-token\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:44:07.207674 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.207686    2560 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-cacerts\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:44:07.207932 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.207694    2560 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-kubeconfig\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:44:07.207932 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.207702    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-979w8\" (UniqueName: \"kubernetes.io/projected/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-kube-api-access-979w8\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:44:07.207932 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.207711    2560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9-istio-csr-ca-configmap\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:44:07.968681 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:07.968649    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:44:08.039267 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.039227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg" event={"ID":"409f8a8b-40fa-4720-8f44-ec4a82ea2ee9","Type":"ContainerDied","Data":"66179ddf63393aa6520aca0a0bd46d092b4f5d24d3f0ee7403b0cbd2f7250448"}
Jul 29 18:44:08.039451 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.039297    2560 scope.go:117] "RemoveContainer" containerID="631b527af9f68d162a23e446ae0a962714f1fa933453aaab568a8f33752f4d01"
Jul 29 18:44:08.039451 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.039240    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"
Jul 29 18:44:08.041093 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.041073    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6c559f7cd9-f6429"]
Jul 29 18:44:08.041411 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.041398    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" containerName="discovery"
Jul 29 18:44:08.041461 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.041413    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" containerName="discovery"
Jul 29 18:44:08.041500 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.041483    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" containerName="discovery"
Jul 29 18:44:08.043457 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.043438    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c559f7cd9-f6429"
Jul 29 18:44:08.046763 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.046721    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 29 18:44:08.046889 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.046765    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 29 18:44:08.046889 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.046784    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-m8jb2\""
Jul 29 18:44:08.046889 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.046795    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 29 18:44:08.051051 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.051023    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:44:08.053087 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.053047    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6c559f7cd9-f6429"]
Jul 29 18:44:08.062892 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.062858    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"]
Jul 29 18:44:08.065762 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.065745    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"
Jul 29 18:44:08.067885 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.067866    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 29 18:44:08.068419 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.068404    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-25fc7\""
Jul 29 18:44:08.077876 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.077853    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"]
Jul 29 18:44:08.096904 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.096873    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"]
Jul 29 18:44:08.098673 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.098648    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-8445dbf85d-wsrf9"]
Jul 29 18:44:08.100783 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.100767    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g8vpg"]
Jul 29 18:44:08.100866 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.100842    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-8445dbf85d-wsrf9"
Jul 29 18:44:08.103308 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.103288    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 29 18:44:08.103460 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.103439    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-95bkx\""
Jul 29 18:44:08.103903 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.103883    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-8445dbf85d-wsrf9"]
Jul 29 18:44:08.114370 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.114349    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8484805b-b032-4b5e-be09-f1abdbb7661e-cert\") pod \"kserve-controller-manager-6c559f7cd9-f6429\" (UID: \"8484805b-b032-4b5e-be09-f1abdbb7661e\") " pod="kserve/kserve-controller-manager-6c559f7cd9-f6429"
Jul 29 18:44:08.114467 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.114387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z2s8\" (UniqueName: \"kubernetes.io/projected/8564439b-a99c-4451-8620-3b37564e8d89-kube-api-access-9z2s8\") pod \"llmisvc-controller-manager-6f6f7d67d7-r2nvb\" (UID: \"8564439b-a99c-4451-8620-3b37564e8d89\") " pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"
Jul 29 18:44:08.114518 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.114462    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d56h\" (UniqueName: \"kubernetes.io/projected/484a885d-83b6-4779-8e99-a6a44a0212ce-kube-api-access-9d56h\") pod \"seaweedfs-8445dbf85d-wsrf9\" (UID: \"484a885d-83b6-4779-8e99-a6a44a0212ce\") " pod="kserve/seaweedfs-8445dbf85d-wsrf9"
Jul 29 18:44:08.114518 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.114502    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhpgr\" (UniqueName: \"kubernetes.io/projected/8484805b-b032-4b5e-be09-f1abdbb7661e-kube-api-access-dhpgr\") pod \"kserve-controller-manager-6c559f7cd9-f6429\" (UID: \"8484805b-b032-4b5e-be09-f1abdbb7661e\") " pod="kserve/kserve-controller-manager-6c559f7cd9-f6429"
Jul 29 18:44:08.114597 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.114554    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8564439b-a99c-4451-8620-3b37564e8d89-cert\") pod \"llmisvc-controller-manager-6f6f7d67d7-r2nvb\" (UID: \"8564439b-a99c-4451-8620-3b37564e8d89\") " pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"
Jul 29 18:44:08.114597 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.114584    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/484a885d-83b6-4779-8e99-a6a44a0212ce-data\") pod \"seaweedfs-8445dbf85d-wsrf9\" (UID: \"484a885d-83b6-4779-8e99-a6a44a0212ce\") " pod="kserve/seaweedfs-8445dbf85d-wsrf9"
Jul 29 18:44:08.142840 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.142809    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:44:08.215504 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.215473    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8564439b-a99c-4451-8620-3b37564e8d89-cert\") pod \"llmisvc-controller-manager-6f6f7d67d7-r2nvb\" (UID: \"8564439b-a99c-4451-8620-3b37564e8d89\") " pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"
Jul 29 18:44:08.215649 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.215507    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/484a885d-83b6-4779-8e99-a6a44a0212ce-data\") pod \"seaweedfs-8445dbf85d-wsrf9\" (UID: \"484a885d-83b6-4779-8e99-a6a44a0212ce\") " pod="kserve/seaweedfs-8445dbf85d-wsrf9"
Jul 29 18:44:08.215649 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.215542    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8484805b-b032-4b5e-be09-f1abdbb7661e-cert\") pod \"kserve-controller-manager-6c559f7cd9-f6429\" (UID: \"8484805b-b032-4b5e-be09-f1abdbb7661e\") " pod="kserve/kserve-controller-manager-6c559f7cd9-f6429"
Jul 29 18:44:08.215649 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:44:08.215619    2560 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 29 18:44:08.215649 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:44:08.215627    2560 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 29 18:44:08.215805 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:44:08.215666    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8484805b-b032-4b5e-be09-f1abdbb7661e-cert podName:8484805b-b032-4b5e-be09-f1abdbb7661e nodeName:}" failed. No retries permitted until 2026-07-29 18:44:08.715651001 +0000 UTC m=+574.609651381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8484805b-b032-4b5e-be09-f1abdbb7661e-cert") pod "kserve-controller-manager-6c559f7cd9-f6429" (UID: "8484805b-b032-4b5e-be09-f1abdbb7661e") : secret "kserve-webhook-server-cert" not found
Jul 29 18:44:08.215805 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:44:08.215689    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8564439b-a99c-4451-8620-3b37564e8d89-cert podName:8564439b-a99c-4451-8620-3b37564e8d89 nodeName:}" failed. No retries permitted until 2026-07-29 18:44:08.715670775 +0000 UTC m=+574.609671157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8564439b-a99c-4451-8620-3b37564e8d89-cert") pod "llmisvc-controller-manager-6f6f7d67d7-r2nvb" (UID: "8564439b-a99c-4451-8620-3b37564e8d89") : secret "llmisvc-webhook-server-cert" not found
Jul 29 18:44:08.215805 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.215719    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z2s8\" (UniqueName: \"kubernetes.io/projected/8564439b-a99c-4451-8620-3b37564e8d89-kube-api-access-9z2s8\") pod \"llmisvc-controller-manager-6f6f7d67d7-r2nvb\" (UID: \"8564439b-a99c-4451-8620-3b37564e8d89\") " pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"
Jul 29 18:44:08.215805 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.215784    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d56h\" (UniqueName: \"kubernetes.io/projected/484a885d-83b6-4779-8e99-a6a44a0212ce-kube-api-access-9d56h\") pod \"seaweedfs-8445dbf85d-wsrf9\" (UID: \"484a885d-83b6-4779-8e99-a6a44a0212ce\") " pod="kserve/seaweedfs-8445dbf85d-wsrf9"
Jul 29 18:44:08.215960 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.215813    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhpgr\" (UniqueName: \"kubernetes.io/projected/8484805b-b032-4b5e-be09-f1abdbb7661e-kube-api-access-dhpgr\") pod \"kserve-controller-manager-6c559f7cd9-f6429\" (UID: \"8484805b-b032-4b5e-be09-f1abdbb7661e\") " pod="kserve/kserve-controller-manager-6c559f7cd9-f6429"
Jul 29 18:44:08.215960 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.215900    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/484a885d-83b6-4779-8e99-a6a44a0212ce-data\") pod \"seaweedfs-8445dbf85d-wsrf9\" (UID: \"484a885d-83b6-4779-8e99-a6a44a0212ce\") " pod="kserve/seaweedfs-8445dbf85d-wsrf9"
Jul 29 18:44:08.226617 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.226559    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d56h\" (UniqueName: \"kubernetes.io/projected/484a885d-83b6-4779-8e99-a6a44a0212ce-kube-api-access-9d56h\") pod \"seaweedfs-8445dbf85d-wsrf9\" (UID: \"484a885d-83b6-4779-8e99-a6a44a0212ce\") " pod="kserve/seaweedfs-8445dbf85d-wsrf9"
Jul 29 18:44:08.226617 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.226578    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhpgr\" (UniqueName: \"kubernetes.io/projected/8484805b-b032-4b5e-be09-f1abdbb7661e-kube-api-access-dhpgr\") pod \"kserve-controller-manager-6c559f7cd9-f6429\" (UID: \"8484805b-b032-4b5e-be09-f1abdbb7661e\") " pod="kserve/kserve-controller-manager-6c559f7cd9-f6429"
Jul 29 18:44:08.226763 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.226691    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z2s8\" (UniqueName: \"kubernetes.io/projected/8564439b-a99c-4451-8620-3b37564e8d89-kube-api-access-9z2s8\") pod \"llmisvc-controller-manager-6f6f7d67d7-r2nvb\" (UID: \"8564439b-a99c-4451-8620-3b37564e8d89\") " pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"
Jul 29 18:44:08.344165 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.344133    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:44:08.412580 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.412550    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-8445dbf85d-wsrf9"
Jul 29 18:44:08.508431 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.508399    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:44:08.541373 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.541348    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-8445dbf85d-wsrf9"]
Jul 29 18:44:08.543041 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:44:08.543015    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484a885d_83b6_4779_8e99_a6a44a0212ce.slice/crio-fd7d857db6e414f8779f2a3b26f318f740db2168c7f2a6c454dfa5fb43037dcf WatchSource:0}: Error finding container fd7d857db6e414f8779f2a3b26f318f740db2168c7f2a6c454dfa5fb43037dcf: Status 404 returned error can't find the container with id fd7d857db6e414f8779f2a3b26f318f740db2168c7f2a6c454dfa5fb43037dcf
Jul 29 18:44:08.686919 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.686885    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="409f8a8b-40fa-4720-8f44-ec4a82ea2ee9" path="/var/lib/kubelet/pods/409f8a8b-40fa-4720-8f44-ec4a82ea2ee9/volumes"
Jul 29 18:44:08.722315 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.722272    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8564439b-a99c-4451-8620-3b37564e8d89-cert\") pod \"llmisvc-controller-manager-6f6f7d67d7-r2nvb\" (UID: \"8564439b-a99c-4451-8620-3b37564e8d89\") " pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"
Jul 29 18:44:08.722473 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.722408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8484805b-b032-4b5e-be09-f1abdbb7661e-cert\") pod \"kserve-controller-manager-6c559f7cd9-f6429\" (UID: \"8484805b-b032-4b5e-be09-f1abdbb7661e\") " pod="kserve/kserve-controller-manager-6c559f7cd9-f6429"
Jul 29 18:44:08.724833 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.724808    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8564439b-a99c-4451-8620-3b37564e8d89-cert\") pod \"llmisvc-controller-manager-6f6f7d67d7-r2nvb\" (UID: \"8564439b-a99c-4451-8620-3b37564e8d89\") " pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"
Jul 29 18:44:08.724965 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.724933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8484805b-b032-4b5e-be09-f1abdbb7661e-cert\") pod \"kserve-controller-manager-6c559f7cd9-f6429\" (UID: \"8484805b-b032-4b5e-be09-f1abdbb7661e\") " pod="kserve/kserve-controller-manager-6c559f7cd9-f6429"
Jul 29 18:44:08.755389 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.755317    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:44:08.852617 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.852589    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:44:08.957062 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.957030    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c559f7cd9-f6429"
Jul 29 18:44:08.975490 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:08.975461    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"
Jul 29 18:44:09.048921 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:09.048873    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-8445dbf85d-wsrf9" event={"ID":"484a885d-83b6-4779-8e99-a6a44a0212ce","Type":"ContainerStarted","Data":"fd7d857db6e414f8779f2a3b26f318f740db2168c7f2a6c454dfa5fb43037dcf"}
Jul 29 18:44:09.172441 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:09.172408    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6c559f7cd9-f6429"]
Jul 29 18:44:09.174301 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:44:09.174271    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8484805b_b032_4b5e_be09_f1abdbb7661e.slice/crio-b85bb4a7cd795b4c2bfbd3755ab9900f6d389934b13d3372807f793f8ef7db65 WatchSource:0}: Error finding container b85bb4a7cd795b4c2bfbd3755ab9900f6d389934b13d3372807f793f8ef7db65: Status 404 returned error can't find the container with id b85bb4a7cd795b4c2bfbd3755ab9900f6d389934b13d3372807f793f8ef7db65
Jul 29 18:44:09.189796 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:09.189769    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"]
Jul 29 18:44:09.191784 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:44:09.191753    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8564439b_a99c_4451_8620_3b37564e8d89.slice/crio-e188400a4d7a65964c017eeadec16f65cd02ba74a1094d5a14e30a649328a099 WatchSource:0}: Error finding container e188400a4d7a65964c017eeadec16f65cd02ba74a1094d5a14e30a649328a099: Status 404 returned error can't find the container with id e188400a4d7a65964c017eeadec16f65cd02ba74a1094d5a14e30a649328a099
Jul 29 18:44:10.061007 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:10.060967    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c559f7cd9-f6429" event={"ID":"8484805b-b032-4b5e-be09-f1abdbb7661e","Type":"ContainerStarted","Data":"b85bb4a7cd795b4c2bfbd3755ab9900f6d389934b13d3372807f793f8ef7db65"}
Jul 29 18:44:10.062510 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:10.062484    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb" event={"ID":"8564439b-a99c-4451-8620-3b37564e8d89","Type":"ContainerStarted","Data":"e188400a4d7a65964c017eeadec16f65cd02ba74a1094d5a14e30a649328a099"}
Jul 29 18:44:14.089794 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:14.089757    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-8445dbf85d-wsrf9" event={"ID":"484a885d-83b6-4779-8e99-a6a44a0212ce","Type":"ContainerStarted","Data":"334b03f5fe108cf87676f5f7da9bfed7981b598c1341c6addfc39e6c9a60f0e0"}
Jul 29 18:44:14.090286 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:14.089864    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-8445dbf85d-wsrf9"
Jul 29 18:44:14.091276 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:14.091251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c559f7cd9-f6429" event={"ID":"8484805b-b032-4b5e-be09-f1abdbb7661e","Type":"ContainerStarted","Data":"77bb3359a0218687ab1ff99700311c46ea0940c001cd6c1d5cb9903fc39f0f53"}
Jul 29 18:44:14.091424 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:14.091360    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6c559f7cd9-f6429"
Jul 29 18:44:14.092675 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:14.092652    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb" event={"ID":"8564439b-a99c-4451-8620-3b37564e8d89","Type":"ContainerStarted","Data":"75b8e22f293f9a7b8103216bb7eb86aede5c36bb7571b86e2efd7770ef37e259"}
Jul 29 18:44:14.092794 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:14.092773    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"
Jul 29 18:44:14.107809 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:14.107765    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-8445dbf85d-wsrf9" podStartSLOduration=0.915403223 podStartE2EDuration="6.107752795s" totalImagesPullingTime="5.192349572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:44:14.106652634 +0000 UTC m=+580.000653053" watchObservedRunningTime="2026-07-29 18:44:14.107752795 +0000 UTC m=+580.001753196"
Jul 29 18:44:14.123758 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:14.123714    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb" podStartSLOduration=1.659670369 podStartE2EDuration="6.123702043s" totalImagesPullingTime="4.464031674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:44:14.123522945 +0000 UTC m=+580.017523365" watchObservedRunningTime="2026-07-29 18:44:14.123702043 +0000 UTC m=+580.017702422"
Jul 29 18:44:14.140894 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:14.140848    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6c559f7cd9-f6429" podStartSLOduration=1.6592081570000001 podStartE2EDuration="6.140836587s" totalImagesPullingTime="4.48162843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:44:14.139349509 +0000 UTC m=+580.033350036" watchObservedRunningTime="2026-07-29 18:44:14.140836587 +0000 UTC m=+580.034836987"
Jul 29 18:44:20.100000 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:20.099967    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-8445dbf85d-wsrf9"
Jul 29 18:44:20.815551 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:20.815515    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:44:20.915921 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:20.915885    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:44:34.494073 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:34.494046    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 18:44:34.494518 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:34.494046    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 18:44:45.099908 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:45.099874    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"
Jul 29 18:44:45.102846 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:45.102825    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6c559f7cd9-f6429"
Jul 29 18:44:46.277713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.277676    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6c559f7cd9-f6429"]
Jul 29 18:44:46.292455 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.277869    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6c559f7cd9-f6429" podUID="8484805b-b032-4b5e-be09-f1abdbb7661e" containerName="manager" containerID="cri-o://77bb3359a0218687ab1ff99700311c46ea0940c001cd6c1d5cb9903fc39f0f53" gracePeriod=10
Jul 29 18:44:46.301814 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.301791    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6c559f7cd9-vrx65"]
Jul 29 18:44:46.318355 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.318329    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6c559f7cd9-vrx65"]
Jul 29 18:44:46.318480 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.318442    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c559f7cd9-vrx65"
Jul 29 18:44:46.449606 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.449575    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2fbd76c-54bc-4252-af02-7f32764e0cdf-cert\") pod \"kserve-controller-manager-6c559f7cd9-vrx65\" (UID: \"a2fbd76c-54bc-4252-af02-7f32764e0cdf\") " pod="kserve/kserve-controller-manager-6c559f7cd9-vrx65"
Jul 29 18:44:46.449766 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.449616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnsjw\" (UniqueName: \"kubernetes.io/projected/a2fbd76c-54bc-4252-af02-7f32764e0cdf-kube-api-access-fnsjw\") pod \"kserve-controller-manager-6c559f7cd9-vrx65\" (UID: \"a2fbd76c-54bc-4252-af02-7f32764e0cdf\") " pod="kserve/kserve-controller-manager-6c559f7cd9-vrx65"
Jul 29 18:44:46.544901 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.544877    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c559f7cd9-f6429"
Jul 29 18:44:46.550957 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.550932    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2fbd76c-54bc-4252-af02-7f32764e0cdf-cert\") pod \"kserve-controller-manager-6c559f7cd9-vrx65\" (UID: \"a2fbd76c-54bc-4252-af02-7f32764e0cdf\") " pod="kserve/kserve-controller-manager-6c559f7cd9-vrx65"
Jul 29 18:44:46.551070 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.550965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnsjw\" (UniqueName: \"kubernetes.io/projected/a2fbd76c-54bc-4252-af02-7f32764e0cdf-kube-api-access-fnsjw\") pod \"kserve-controller-manager-6c559f7cd9-vrx65\" (UID: \"a2fbd76c-54bc-4252-af02-7f32764e0cdf\") " pod="kserve/kserve-controller-manager-6c559f7cd9-vrx65"
Jul 29 18:44:46.553263 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.553244    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2fbd76c-54bc-4252-af02-7f32764e0cdf-cert\") pod \"kserve-controller-manager-6c559f7cd9-vrx65\" (UID: \"a2fbd76c-54bc-4252-af02-7f32764e0cdf\") " pod="kserve/kserve-controller-manager-6c559f7cd9-vrx65"
Jul 29 18:44:46.559810 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.559788    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnsjw\" (UniqueName: \"kubernetes.io/projected/a2fbd76c-54bc-4252-af02-7f32764e0cdf-kube-api-access-fnsjw\") pod \"kserve-controller-manager-6c559f7cd9-vrx65\" (UID: \"a2fbd76c-54bc-4252-af02-7f32764e0cdf\") " pod="kserve/kserve-controller-manager-6c559f7cd9-vrx65"
Jul 29 18:44:46.651795 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.651762    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8484805b-b032-4b5e-be09-f1abdbb7661e-cert\") pod \"8484805b-b032-4b5e-be09-f1abdbb7661e\" (UID: \"8484805b-b032-4b5e-be09-f1abdbb7661e\") "
Jul 29 18:44:46.651946 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.651807    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhpgr\" (UniqueName: \"kubernetes.io/projected/8484805b-b032-4b5e-be09-f1abdbb7661e-kube-api-access-dhpgr\") pod \"8484805b-b032-4b5e-be09-f1abdbb7661e\" (UID: \"8484805b-b032-4b5e-be09-f1abdbb7661e\") "
Jul 29 18:44:46.654014 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.653986    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8484805b-b032-4b5e-be09-f1abdbb7661e-kube-api-access-dhpgr" (OuterVolumeSpecName: "kube-api-access-dhpgr") pod "8484805b-b032-4b5e-be09-f1abdbb7661e" (UID: "8484805b-b032-4b5e-be09-f1abdbb7661e"). InnerVolumeSpecName "kube-api-access-dhpgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:44:46.654117 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.654009    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8484805b-b032-4b5e-be09-f1abdbb7661e-cert" (OuterVolumeSpecName: "cert") pod "8484805b-b032-4b5e-be09-f1abdbb7661e" (UID: "8484805b-b032-4b5e-be09-f1abdbb7661e"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:44:46.667793 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.667763    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c559f7cd9-vrx65"
Jul 29 18:44:46.752915 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.752878    2560 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8484805b-b032-4b5e-be09-f1abdbb7661e-cert\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:44:46.752915 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.752915    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhpgr\" (UniqueName: \"kubernetes.io/projected/8484805b-b032-4b5e-be09-f1abdbb7661e-kube-api-access-dhpgr\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:44:46.797192 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:46.797167    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6c559f7cd9-vrx65"]
Jul 29 18:44:46.799456 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:44:46.799422    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fbd76c_54bc_4252_af02_7f32764e0cdf.slice/crio-ef199e0a212f0af6e241c588d5d596c3c38595901aad2f85e24f0feacd3c5c4f WatchSource:0}: Error finding container ef199e0a212f0af6e241c588d5d596c3c38595901aad2f85e24f0feacd3c5c4f: Status 404 returned error can't find the container with id ef199e0a212f0af6e241c588d5d596c3c38595901aad2f85e24f0feacd3c5c4f
Jul 29 18:44:47.253615 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:47.253580    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c559f7cd9-vrx65" event={"ID":"a2fbd76c-54bc-4252-af02-7f32764e0cdf","Type":"ContainerStarted","Data":"ef199e0a212f0af6e241c588d5d596c3c38595901aad2f85e24f0feacd3c5c4f"}
Jul 29 18:44:47.254758 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:47.254722    2560 generic.go:356] "Generic (PLEG): container finished" podID="8484805b-b032-4b5e-be09-f1abdbb7661e" containerID="77bb3359a0218687ab1ff99700311c46ea0940c001cd6c1d5cb9903fc39f0f53" exitCode=0
Jul 29 18:44:47.254862 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:47.254788    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c559f7cd9-f6429"
Jul 29 18:44:47.254862 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:47.254813    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c559f7cd9-f6429" event={"ID":"8484805b-b032-4b5e-be09-f1abdbb7661e","Type":"ContainerDied","Data":"77bb3359a0218687ab1ff99700311c46ea0940c001cd6c1d5cb9903fc39f0f53"}
Jul 29 18:44:47.254930 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:47.254873    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c559f7cd9-f6429" event={"ID":"8484805b-b032-4b5e-be09-f1abdbb7661e","Type":"ContainerDied","Data":"b85bb4a7cd795b4c2bfbd3755ab9900f6d389934b13d3372807f793f8ef7db65"}
Jul 29 18:44:47.254930 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:47.254897    2560 scope.go:117] "RemoveContainer" containerID="77bb3359a0218687ab1ff99700311c46ea0940c001cd6c1d5cb9903fc39f0f53"
Jul 29 18:44:47.265783 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:47.265124    2560 scope.go:117] "RemoveContainer" containerID="77bb3359a0218687ab1ff99700311c46ea0940c001cd6c1d5cb9903fc39f0f53"
Jul 29 18:44:47.265783 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:44:47.265431    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77bb3359a0218687ab1ff99700311c46ea0940c001cd6c1d5cb9903fc39f0f53\": container with ID starting with 77bb3359a0218687ab1ff99700311c46ea0940c001cd6c1d5cb9903fc39f0f53 not found: ID does not exist" containerID="77bb3359a0218687ab1ff99700311c46ea0940c001cd6c1d5cb9903fc39f0f53"
Jul 29 18:44:47.265783 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:47.265482    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77bb3359a0218687ab1ff99700311c46ea0940c001cd6c1d5cb9903fc39f0f53"} err="failed to get container status \"77bb3359a0218687ab1ff99700311c46ea0940c001cd6c1d5cb9903fc39f0f53\": rpc error: code = NotFound desc = could not find container \"77bb3359a0218687ab1ff99700311c46ea0940c001cd6c1d5cb9903fc39f0f53\": container with ID starting with 77bb3359a0218687ab1ff99700311c46ea0940c001cd6c1d5cb9903fc39f0f53 not found: ID does not exist"
Jul 29 18:44:47.272079 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:47.272054    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6c559f7cd9-f6429"]
Jul 29 18:44:47.276310 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:47.276290    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6c559f7cd9-f6429"]
Jul 29 18:44:48.260676 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:48.260637    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c559f7cd9-vrx65" event={"ID":"a2fbd76c-54bc-4252-af02-7f32764e0cdf","Type":"ContainerStarted","Data":"a002d5124bbffd30281ba6e4395edd4a6ae8d0e3376eb740f3972209258709be"}
Jul 29 18:44:48.261156 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:48.260710    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6c559f7cd9-vrx65"
Jul 29 18:44:48.278405 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:48.278360    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6c559f7cd9-vrx65" podStartSLOduration=1.8332837880000001 podStartE2EDuration="2.278344041s" totalImagesPullingTime="445.060253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:44:48.277298563 +0000 UTC m=+614.171298975" watchObservedRunningTime="2026-07-29 18:44:48.278344041 +0000 UTC m=+614.172344441"
Jul 29 18:44:48.688164 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:48.688129    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8484805b-b032-4b5e-be09-f1abdbb7661e" path="/var/lib/kubelet/pods/8484805b-b032-4b5e-be09-f1abdbb7661e/volumes"
Jul 29 18:44:51.806591 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:44:51.806562    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:45:19.271494 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:19.271456    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6c559f7cd9-vrx65"
Jul 29 18:45:20.492341 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.492307    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:45:20.532452 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.532421    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-h6xdl"]
Jul 29 18:45:20.532756 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.532744    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8484805b-b032-4b5e-be09-f1abdbb7661e" containerName="manager"
Jul 29 18:45:20.532813 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.532757    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8484805b-b032-4b5e-be09-f1abdbb7661e" containerName="manager"
Jul 29 18:45:20.532850 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.532817    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8484805b-b032-4b5e-be09-f1abdbb7661e" containerName="manager"
Jul 29 18:45:20.534678 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.534659    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-h6xdl"
Jul 29 18:45:20.537119 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.537096    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-2gbh7\""
Jul 29 18:45:20.537233 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.537212    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 29 18:45:20.541965 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.541945    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zwkth"]
Jul 29 18:45:20.544128 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.544111    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-zwkth"
Jul 29 18:45:20.546489 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.546453    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-7w95f\""
Jul 29 18:45:20.546489 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.546473    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 29 18:45:20.548250 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.548226    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-h6xdl"]
Jul 29 18:45:20.555411 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.555388    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zwkth"]
Jul 29 18:45:20.564003 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.563980    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:45:20.602451 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.602424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860953da-989c-4afb-8389-7a123345af55-tls-certs\") pod \"model-serving-api-7b954ffc95-h6xdl\" (UID: \"860953da-989c-4afb-8389-7a123345af55\") " pod="kserve/model-serving-api-7b954ffc95-h6xdl"
Jul 29 18:45:20.602793 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.602479    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5crj\" (UniqueName: \"kubernetes.io/projected/860953da-989c-4afb-8389-7a123345af55-kube-api-access-j5crj\") pod \"model-serving-api-7b954ffc95-h6xdl\" (UID: \"860953da-989c-4afb-8389-7a123345af55\") " pod="kserve/model-serving-api-7b954ffc95-h6xdl"
Jul 29 18:45:20.602793 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.602562    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98de5b69-59a1-4737-8775-8c4cbb27c291-cert\") pod \"odh-model-controller-75d9cdbddf-zwkth\" (UID: \"98de5b69-59a1-4737-8775-8c4cbb27c291\") " pod="kserve/odh-model-controller-75d9cdbddf-zwkth"
Jul 29 18:45:20.602793 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.602599    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b52sn\" (UniqueName: \"kubernetes.io/projected/98de5b69-59a1-4737-8775-8c4cbb27c291-kube-api-access-b52sn\") pod \"odh-model-controller-75d9cdbddf-zwkth\" (UID: \"98de5b69-59a1-4737-8775-8c4cbb27c291\") " pod="kserve/odh-model-controller-75d9cdbddf-zwkth"
Jul 29 18:45:20.645166 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.645136    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:45:20.704005 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.703979    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860953da-989c-4afb-8389-7a123345af55-tls-certs\") pod \"model-serving-api-7b954ffc95-h6xdl\" (UID: \"860953da-989c-4afb-8389-7a123345af55\") " pod="kserve/model-serving-api-7b954ffc95-h6xdl"
Jul 29 18:45:20.704155 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.704030    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5crj\" (UniqueName: \"kubernetes.io/projected/860953da-989c-4afb-8389-7a123345af55-kube-api-access-j5crj\") pod \"model-serving-api-7b954ffc95-h6xdl\" (UID: \"860953da-989c-4afb-8389-7a123345af55\") " pod="kserve/model-serving-api-7b954ffc95-h6xdl"
Jul 29 18:45:20.704155 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.704058    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98de5b69-59a1-4737-8775-8c4cbb27c291-cert\") pod \"odh-model-controller-75d9cdbddf-zwkth\" (UID: \"98de5b69-59a1-4737-8775-8c4cbb27c291\") " pod="kserve/odh-model-controller-75d9cdbddf-zwkth"
Jul 29 18:45:20.704155 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.704079    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b52sn\" (UniqueName: \"kubernetes.io/projected/98de5b69-59a1-4737-8775-8c4cbb27c291-kube-api-access-b52sn\") pod \"odh-model-controller-75d9cdbddf-zwkth\" (UID: \"98de5b69-59a1-4737-8775-8c4cbb27c291\") " pod="kserve/odh-model-controller-75d9cdbddf-zwkth"
Jul 29 18:45:20.706378 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.706356    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98de5b69-59a1-4737-8775-8c4cbb27c291-cert\") pod \"odh-model-controller-75d9cdbddf-zwkth\" (UID: \"98de5b69-59a1-4737-8775-8c4cbb27c291\") " pod="kserve/odh-model-controller-75d9cdbddf-zwkth"
Jul 29 18:45:20.706474 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.706463    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860953da-989c-4afb-8389-7a123345af55-tls-certs\") pod \"model-serving-api-7b954ffc95-h6xdl\" (UID: \"860953da-989c-4afb-8389-7a123345af55\") " pod="kserve/model-serving-api-7b954ffc95-h6xdl"
Jul 29 18:45:20.712311 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.712280    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5crj\" (UniqueName: \"kubernetes.io/projected/860953da-989c-4afb-8389-7a123345af55-kube-api-access-j5crj\") pod \"model-serving-api-7b954ffc95-h6xdl\" (UID: \"860953da-989c-4afb-8389-7a123345af55\") " pod="kserve/model-serving-api-7b954ffc95-h6xdl"
Jul 29 18:45:20.712469 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.712450    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b52sn\" (UniqueName: \"kubernetes.io/projected/98de5b69-59a1-4737-8775-8c4cbb27c291-kube-api-access-b52sn\") pod \"odh-model-controller-75d9cdbddf-zwkth\" (UID: \"98de5b69-59a1-4737-8775-8c4cbb27c291\") " pod="kserve/odh-model-controller-75d9cdbddf-zwkth"
Jul 29 18:45:20.748281 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.748222    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:45:20.845456 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.845423    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-h6xdl"
Jul 29 18:45:20.854720 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.854696    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-zwkth"
Jul 29 18:45:20.994867 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.994839    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-h6xdl"]
Jul 29 18:45:20.996778 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:45:20.996749    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860953da_989c_4afb_8389_7a123345af55.slice/crio-0aba4883fc6d07e8bca9c3656fd82998d35e3cb8ffe6257dce0352b0d84f2858 WatchSource:0}: Error finding container 0aba4883fc6d07e8bca9c3656fd82998d35e3cb8ffe6257dce0352b0d84f2858: Status 404 returned error can't find the container with id 0aba4883fc6d07e8bca9c3656fd82998d35e3cb8ffe6257dce0352b0d84f2858
Jul 29 18:45:20.998781 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:20.998762    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 18:45:21.018921 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:21.018889    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zwkth"]
Jul 29 18:45:21.020894 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:45:21.020869    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98de5b69_59a1_4737_8775_8c4cbb27c291.slice/crio-56fec7ad110e849536b8503eba27a81c6fe7249ddc1f14026e1f44368af2163e WatchSource:0}: Error finding container 56fec7ad110e849536b8503eba27a81c6fe7249ddc1f14026e1f44368af2163e: Status 404 returned error can't find the container with id 56fec7ad110e849536b8503eba27a81c6fe7249ddc1f14026e1f44368af2163e
Jul 29 18:45:21.226417 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:21.226382    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:45:21.326622 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:21.326590    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:45:21.395005 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:21.394973    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:45:21.423441 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:21.423399    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-h6xdl" event={"ID":"860953da-989c-4afb-8389-7a123345af55","Type":"ContainerStarted","Data":"0aba4883fc6d07e8bca9c3656fd82998d35e3cb8ffe6257dce0352b0d84f2858"}
Jul 29 18:45:21.424468 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:21.424445    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-zwkth" event={"ID":"98de5b69-59a1-4737-8775-8c4cbb27c291","Type":"ContainerStarted","Data":"56fec7ad110e849536b8503eba27a81c6fe7249ddc1f14026e1f44368af2163e"}
Jul 29 18:45:24.443632 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:24.443592    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-h6xdl" event={"ID":"860953da-989c-4afb-8389-7a123345af55","Type":"ContainerStarted","Data":"8767206d87f58ea959272022e4a2e07e3c86852deef59c75caff5aa6f6e26683"}
Jul 29 18:45:24.444147 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:24.443695    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-h6xdl"
Jul 29 18:45:24.464940 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:24.464892    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-h6xdl" podStartSLOduration=1.816502568 podStartE2EDuration="4.46487086s" totalImagesPullingTime="2.648368292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:45:24.461535076 +0000 UTC m=+650.355535478" watchObservedRunningTime="2026-07-29 18:45:24.46487086 +0000 UTC m=+650.358871261"
Jul 29 18:45:26.456512 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:26.456471    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-zwkth" event={"ID":"98de5b69-59a1-4737-8775-8c4cbb27c291","Type":"ContainerStarted","Data":"bd1c9baba3c7b17050abf3c0a38cbb00a3fe234ecebaa5b8de9da0568b79a2a8"}
Jul 29 18:45:26.456911 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:26.456623    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-zwkth"
Jul 29 18:45:26.473846 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:26.473788    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-zwkth" podStartSLOduration=1.676374133 podStartE2EDuration="6.473770324s" totalImagesPullingTime="4.797396191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:45:26.47358709 +0000 UTC m=+652.367587512" watchObservedRunningTime="2026-07-29 18:45:26.473770324 +0000 UTC m=+652.367770725"
Jul 29 18:45:35.453530 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:35.453496    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-h6xdl"
Jul 29 18:45:37.463250 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:37.463213    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-zwkth"
Jul 29 18:45:38.318052 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:38.318002    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-fnjkq"]
Jul 29 18:45:38.321556 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:38.321533    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fnjkq"
Jul 29 18:45:38.331390 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:38.331366    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-fnjkq"]
Jul 29 18:45:38.458802 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:38.458764    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cfdj\" (UniqueName: \"kubernetes.io/projected/a1585ad1-7b3b-4006-b4c0-2b8162c20613-kube-api-access-5cfdj\") pod \"s3-init-fnjkq\" (UID: \"a1585ad1-7b3b-4006-b4c0-2b8162c20613\") " pod="kserve/s3-init-fnjkq"
Jul 29 18:45:38.559567 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:38.559517    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cfdj\" (UniqueName: \"kubernetes.io/projected/a1585ad1-7b3b-4006-b4c0-2b8162c20613-kube-api-access-5cfdj\") pod \"s3-init-fnjkq\" (UID: \"a1585ad1-7b3b-4006-b4c0-2b8162c20613\") " pod="kserve/s3-init-fnjkq"
Jul 29 18:45:38.569304 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:38.569238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cfdj\" (UniqueName: \"kubernetes.io/projected/a1585ad1-7b3b-4006-b4c0-2b8162c20613-kube-api-access-5cfdj\") pod \"s3-init-fnjkq\" (UID: \"a1585ad1-7b3b-4006-b4c0-2b8162c20613\") " pod="kserve/s3-init-fnjkq"
Jul 29 18:45:38.630898 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:38.630866    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fnjkq"
Jul 29 18:45:38.765021 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:38.764996    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-fnjkq"]
Jul 29 18:45:38.766952 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:45:38.766926    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1585ad1_7b3b_4006_b4c0_2b8162c20613.slice/crio-c80172341031642f94fb6f769d371fb999f349a1a8319d4403539b826015c856 WatchSource:0}: Error finding container c80172341031642f94fb6f769d371fb999f349a1a8319d4403539b826015c856: Status 404 returned error can't find the container with id c80172341031642f94fb6f769d371fb999f349a1a8319d4403539b826015c856
Jul 29 18:45:39.531267 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:39.531190    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fnjkq" event={"ID":"a1585ad1-7b3b-4006-b4c0-2b8162c20613","Type":"ContainerStarted","Data":"c80172341031642f94fb6f769d371fb999f349a1a8319d4403539b826015c856"}
Jul 29 18:45:43.559343 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:43.559311    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fnjkq" event={"ID":"a1585ad1-7b3b-4006-b4c0-2b8162c20613","Type":"ContainerStarted","Data":"f2f8105a948de2bfe9ed2b624e426ead1fffe224c920449c355e4e6c31fcbd36"}
Jul 29 18:45:43.575768 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:43.575715    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-fnjkq" podStartSLOduration=1.1668834160000001 podStartE2EDuration="5.575699327s" totalImagesPullingTime="4.408815911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:45:43.574145498 +0000 UTC m=+669.468145900" watchObservedRunningTime="2026-07-29 18:45:43.575699327 +0000 UTC m=+669.469699728"
Jul 29 18:45:46.181677 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:45:46.181645    2560 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1585ad1_7b3b_4006_b4c0_2b8162c20613.slice/crio-f2f8105a948de2bfe9ed2b624e426ead1fffe224c920449c355e4e6c31fcbd36.scope\": RecentStats: unable to find data in memory cache]"
Jul 29 18:45:46.576590 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:46.576556    2560 generic.go:356] "Generic (PLEG): container finished" podID="a1585ad1-7b3b-4006-b4c0-2b8162c20613" containerID="f2f8105a948de2bfe9ed2b624e426ead1fffe224c920449c355e4e6c31fcbd36" exitCode=0
Jul 29 18:45:46.576750 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:46.576629    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fnjkq" event={"ID":"a1585ad1-7b3b-4006-b4c0-2b8162c20613","Type":"ContainerDied","Data":"f2f8105a948de2bfe9ed2b624e426ead1fffe224c920449c355e4e6c31fcbd36"}
Jul 29 18:45:47.707705 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:47.707680    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fnjkq"
Jul 29 18:45:47.840114 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:47.840014    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cfdj\" (UniqueName: \"kubernetes.io/projected/a1585ad1-7b3b-4006-b4c0-2b8162c20613-kube-api-access-5cfdj\") pod \"a1585ad1-7b3b-4006-b4c0-2b8162c20613\" (UID: \"a1585ad1-7b3b-4006-b4c0-2b8162c20613\") "
Jul 29 18:45:47.842098 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:47.842069    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1585ad1-7b3b-4006-b4c0-2b8162c20613-kube-api-access-5cfdj" (OuterVolumeSpecName: "kube-api-access-5cfdj") pod "a1585ad1-7b3b-4006-b4c0-2b8162c20613" (UID: "a1585ad1-7b3b-4006-b4c0-2b8162c20613"). InnerVolumeSpecName "kube-api-access-5cfdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:45:47.941233 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:47.941176    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cfdj\" (UniqueName: \"kubernetes.io/projected/a1585ad1-7b3b-4006-b4c0-2b8162c20613-kube-api-access-5cfdj\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:45:48.587080 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:48.587043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fnjkq" event={"ID":"a1585ad1-7b3b-4006-b4c0-2b8162c20613","Type":"ContainerDied","Data":"c80172341031642f94fb6f769d371fb999f349a1a8319d4403539b826015c856"}
Jul 29 18:45:48.587080 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:48.587069    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fnjkq"
Jul 29 18:45:48.587080 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:48.587084    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c80172341031642f94fb6f769d371fb999f349a1a8319d4403539b826015c856"
Jul 29 18:45:49.073919 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:49.073883    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:45:49.172556 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:49.172521    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:45:58.841418 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:58.841389    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"]
Jul 29 18:45:58.841819 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:58.841712    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1585ad1-7b3b-4006-b4c0-2b8162c20613" containerName="s3-init"
Jul 29 18:45:58.841819 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:58.841724    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1585ad1-7b3b-4006-b4c0-2b8162c20613" containerName="s3-init"
Jul 29 18:45:58.841819 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:58.841815    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1585ad1-7b3b-4006-b4c0-2b8162c20613" containerName="s3-init"
Jul 29 18:45:58.861585 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:58.861560    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"]
Jul 29 18:45:58.861725 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:58.861653    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:58.864138 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:58.864106    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 29 18:45:58.864138 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:58.864106    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 18:45:58.864389 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:58.864174    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-vr9rr\""
Jul 29 18:45:58.864592 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:58.864573    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 18:45:58.869975 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:58.869951    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:45:58.970439 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:58.970405    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:45:59.028950 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.028917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-istio-envoy\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.029138 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.028956    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-credential-socket\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.029138 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.028976    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31372be-e035-44b2-88c2-85e009edb9fc-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.029138 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.029090    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31372be-e035-44b2-88c2-85e009edb9fc-istio-podinfo\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.029138 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.029129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-istio-data\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.029364 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.029147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31372be-e035-44b2-88c2-85e009edb9fc-istio-token\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.029364 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.029243    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-workload-certs\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.029364 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.029270    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-workload-socket\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.029364 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.029301    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkv7h\" (UniqueName: \"kubernetes.io/projected/c31372be-e035-44b2-88c2-85e009edb9fc-kube-api-access-pkv7h\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.129831 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.129749    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31372be-e035-44b2-88c2-85e009edb9fc-istio-podinfo\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.129831 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.129787    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-istio-data\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.129831 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.129805    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31372be-e035-44b2-88c2-85e009edb9fc-istio-token\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.130094 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.129841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-workload-certs\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.130094 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.129861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-workload-socket\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.130094 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.129889    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkv7h\" (UniqueName: \"kubernetes.io/projected/c31372be-e035-44b2-88c2-85e009edb9fc-kube-api-access-pkv7h\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.130094 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.129912    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-istio-envoy\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.130094 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.130076    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-credential-socket\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.130376 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.130105    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31372be-e035-44b2-88c2-85e009edb9fc-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.130376 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.130214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-istio-data\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.130565 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.130522    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-workload-socket\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.130780 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.130752    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-workload-certs\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.130886 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.130872    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-credential-socket\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.131055 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.131033    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31372be-e035-44b2-88c2-85e009edb9fc-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.132822 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.132759    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31372be-e035-44b2-88c2-85e009edb9fc-istio-envoy\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.132945 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.132861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31372be-e035-44b2-88c2-85e009edb9fc-istio-podinfo\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.138410 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.138391    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31372be-e035-44b2-88c2-85e009edb9fc-istio-token\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.138704 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.138673    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkv7h\" (UniqueName: \"kubernetes.io/projected/c31372be-e035-44b2-88c2-85e009edb9fc-kube-api-access-pkv7h\") pod \"router-gateway-1-openshift-default-56f678f445-9wrf5\" (UID: \"c31372be-e035-44b2-88c2-85e009edb9fc\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.172594 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.172558    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:45:59.347546 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.347521    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"]
Jul 29 18:45:59.349598 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:45:59.349562    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31372be_e035_44b2_88c2_85e009edb9fc.slice/crio-7977cdbedc5adb179be318159b9dc842f4412dcc15ba136d03d846ce587edb49 WatchSource:0}: Error finding container 7977cdbedc5adb179be318159b9dc842f4412dcc15ba136d03d846ce587edb49: Status 404 returned error can't find the container with id 7977cdbedc5adb179be318159b9dc842f4412dcc15ba136d03d846ce587edb49
Jul 29 18:45:59.351841 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.351800    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:45:59.351940 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.351903    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:45:59.351940 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.351933    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:45:59.647725 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.647693    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5" event={"ID":"c31372be-e035-44b2-88c2-85e009edb9fc","Type":"ContainerStarted","Data":"be7b3a8b34e05bf9e7ffcf912a02263d570e522fa26bf57cbf5d70c86afd28b6"}
Jul 29 18:45:59.647896 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.647755    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5" event={"ID":"c31372be-e035-44b2-88c2-85e009edb9fc","Type":"ContainerStarted","Data":"7977cdbedc5adb179be318159b9dc842f4412dcc15ba136d03d846ce587edb49"}
Jul 29 18:45:59.668460 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:45:59.668374    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5" podStartSLOduration=1.668359959 podStartE2EDuration="1.668359959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:45:59.666006838 +0000 UTC m=+685.560007240" watchObservedRunningTime="2026-07-29 18:45:59.668359959 +0000 UTC m=+685.562360360"
Jul 29 18:46:00.173729 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:00.173683    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:46:00.179080 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:00.179054    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:46:00.653623 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:00.653591    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:46:00.654578 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:00.654551    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:46:02.705897 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:02.705851    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:46:04.003847 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.003808    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:46:04.291464 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.291377    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"]
Jul 29 18:46:04.306377 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.306331    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.307959 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.307932    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"]
Jul 29 18:46:04.309040 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.309005    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 29 18:46:04.309168 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.309005    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-prmxp\""
Jul 29 18:46:04.309826 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.309793    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2cmt9\""
Jul 29 18:46:04.329402 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.329364    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:46:04.480907 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.480872    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.480907 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.480909    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5335218d-e501-471b-956f-2d578c2e5996-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.481157 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.480935    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt2fw\" (UniqueName: \"kubernetes.io/projected/5335218d-e501-471b-956f-2d578c2e5996-kube-api-access-gt2fw\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.481157 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.480959    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.481157 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.481045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.481157 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.481129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.582602 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.582568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.582798 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.582650    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.582798 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.582681    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5335218d-e501-471b-956f-2d578c2e5996-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.582798 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.582721    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt2fw\" (UniqueName: \"kubernetes.io/projected/5335218d-e501-471b-956f-2d578c2e5996-kube-api-access-gt2fw\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.582798 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.582748    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.583021 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.582803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.583076 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.583029    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.583134 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.583067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.583134 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.583116    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.583302 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.583283    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.585257 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.585236    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5335218d-e501-471b-956f-2d578c2e5996-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.594633 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.594607    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt2fw\" (UniqueName: \"kubernetes.io/projected/5335218d-e501-471b-956f-2d578c2e5996-kube-api-access-gt2fw\") pod \"scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.618787 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.618751    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:04.773971 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:04.773929    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"]
Jul 29 18:46:04.774388 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:46:04.774361    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5335218d_e501_471b_956f_2d578c2e5996.slice/crio-c584da2a39d8787f5262927e7d15d47a2eeafc4c458794a168804e844edbe132 WatchSource:0}: Error finding container c584da2a39d8787f5262927e7d15d47a2eeafc4c458794a168804e844edbe132: Status 404 returned error can't find the container with id c584da2a39d8787f5262927e7d15d47a2eeafc4c458794a168804e844edbe132
Jul 29 18:46:05.682678 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:05.682639    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4" event={"ID":"5335218d-e501-471b-956f-2d578c2e5996","Type":"ContainerStarted","Data":"c584da2a39d8787f5262927e7d15d47a2eeafc4c458794a168804e844edbe132"}
Jul 29 18:46:09.730771 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:09.730729    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4" event={"ID":"5335218d-e501-471b-956f-2d578c2e5996","Type":"ContainerStarted","Data":"8be22d554457b3e909179d864d057a324d9bdf4aec0f365a964762a1af73a247"}
Jul 29 18:46:10.738319 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:10.738219    2560 generic.go:356] "Generic (PLEG): container finished" podID="5335218d-e501-471b-956f-2d578c2e5996" containerID="8be22d554457b3e909179d864d057a324d9bdf4aec0f365a964762a1af73a247" exitCode=0
Jul 29 18:46:10.738722 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:10.738326    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4" event={"ID":"5335218d-e501-471b-956f-2d578c2e5996","Type":"ContainerDied","Data":"8be22d554457b3e909179d864d057a324d9bdf4aec0f365a964762a1af73a247"}
Jul 29 18:46:11.478447 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:11.478400    2560 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 29 18:46:11.478647 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:11.478484    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8" podUID="d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 18:46:12.754602 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:12.754566    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4" event={"ID":"5335218d-e501-471b-956f-2d578c2e5996","Type":"ContainerStarted","Data":"168d266cdf720787e5199ad7af2861b510d1c9fe333806685f1e03b3caf6984b"}
Jul 29 18:46:14.769546 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:14.769512    2560 generic.go:356] "Generic (PLEG): container finished" podID="d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2" containerID="ef3219a439a723f51f2d898c0ef8cb0d26691e32ed6cea05c0434d84c64978be" exitCode=0
Jul 29 18:46:14.770080 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:14.769586    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8" event={"ID":"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2","Type":"ContainerDied","Data":"ef3219a439a723f51f2d898c0ef8cb0d26691e32ed6cea05c0434d84c64978be"}
Jul 29 18:46:14.770080 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:14.770049    2560 scope.go:117] "RemoveContainer" containerID="ef3219a439a723f51f2d898c0ef8cb0d26691e32ed6cea05c0434d84c64978be"
Jul 29 18:46:14.771267 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:14.771228    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:46:14.771404 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:14.771328    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:46:14.771404 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:14.771375    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:46:15.778681 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:15.778646    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8" event={"ID":"d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2","Type":"ContainerStarted","Data":"183ef9ba1bd17993ffef476afb7da61e95913159b106e39fb5db954155571d7e"}
Jul 29 18:46:16.088652 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:16.088613    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:46:16.088889 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:16.088863    2560 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Jul 29 18:46:16.088950 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:16.088923    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8" podUID="d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Jul 29 18:46:17.089140 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:17.089101    2560 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Jul 29 18:46:17.089644 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:17.089160    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8" podUID="d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Jul 29 18:46:18.089293 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:18.089254    2560 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Jul 29 18:46:18.089740 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:18.089325    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8" podUID="d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Jul 29 18:46:19.089705 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:19.089662    2560 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Jul 29 18:46:19.090234 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:19.089749    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8" podUID="d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Jul 29 18:46:20.088839 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:20.088798    2560 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Jul 29 18:46:20.089025 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:20.088870    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8" podUID="d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Jul 29 18:46:21.088860 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:21.088822    2560 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Jul 29 18:46:21.089357 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:21.088884    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8" podUID="d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Jul 29 18:46:22.093523 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:22.093487    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:46:22.094034 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:22.094014    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:46:22.094968 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:22.094939    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-w95c8"
Jul 29 18:46:27.502930 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:27.502841    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:46:37.795948 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:37.795906    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:46:38.659110 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:38.659078    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:46:45.007181 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:45.007141    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4" event={"ID":"5335218d-e501-471b-956f-2d578c2e5996","Type":"ContainerStarted","Data":"a82e3ef225c4718b177f4981cad795b05ae269774c90be373283ea096e3fbff8"}
Jul 29 18:46:45.007658 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:45.007316    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:45.008917 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:45.008886    2560 generic.go:356] "Generic (PLEG): container finished" podID="c31372be-e035-44b2-88c2-85e009edb9fc" containerID="be7b3a8b34e05bf9e7ffcf912a02263d570e522fa26bf57cbf5d70c86afd28b6" exitCode=0
Jul 29 18:46:45.009037 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:45.008943    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5" event={"ID":"c31372be-e035-44b2-88c2-85e009edb9fc","Type":"ContainerDied","Data":"be7b3a8b34e05bf9e7ffcf912a02263d570e522fa26bf57cbf5d70c86afd28b6"}
Jul 29 18:46:45.009321 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:45.009307    2560 scope.go:117] "RemoveContainer" containerID="be7b3a8b34e05bf9e7ffcf912a02263d570e522fa26bf57cbf5d70c86afd28b6"
Jul 29 18:46:45.010337 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:45.010307    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:46:45.010536 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:45.010373    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:45.010536 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:45.010376    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:46:45.010536 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:45.010469    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:46:45.034114 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:45.034070    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4" podStartSLOduration=2.051829204 podStartE2EDuration="41.034056462s" totalImagesPullingTime="38.137862931s" totalInitContainerRuntime="844.364327ms" isStatefulPod=false podCreationTimestamp="2026-07-29 18:46:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:46:45.032740938 +0000 UTC m=+730.926741363" watchObservedRunningTime="2026-07-29 18:46:45.034056462 +0000 UTC m=+730.928056863"
Jul 29 18:46:45.654311 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:45.654276    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:46:46.016135 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:46.016051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5" event={"ID":"c31372be-e035-44b2-88c2-85e009edb9fc","Type":"ContainerStarted","Data":"dce437b3206937304c61c67ea632a3b5c70a0e26dd70a2acbe7736e650f602f7"}
Jul 29 18:46:46.173116 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:46.173082    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:46:46.177437 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:46.177412    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:46:47.021155 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:47.021123    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:46:47.021949 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:47.021920    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:46:48.387296 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:48.387244    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:46:50.795416 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:50.795380    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:46:54.619847 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:54.619807    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:54.620463 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:54.620059    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:54.622078 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:54.622056    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:55.065443 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:55.065413    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:46:59.088629 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:59.088598    2560 generic.go:356] "Generic (PLEG): container finished" podID="c31372be-e035-44b2-88c2-85e009edb9fc" containerID="dce437b3206937304c61c67ea632a3b5c70a0e26dd70a2acbe7736e650f602f7" exitCode=0
Jul 29 18:46:59.089098 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:59.088670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5" event={"ID":"c31372be-e035-44b2-88c2-85e009edb9fc","Type":"ContainerDied","Data":"dce437b3206937304c61c67ea632a3b5c70a0e26dd70a2acbe7736e650f602f7"}
Jul 29 18:46:59.089098 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:59.088735    2560 scope.go:117] "RemoveContainer" containerID="be7b3a8b34e05bf9e7ffcf912a02263d570e522fa26bf57cbf5d70c86afd28b6"
Jul 29 18:46:59.089186 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:46:59.089098    2560 scope.go:117] "RemoveContainer" containerID="dce437b3206937304c61c67ea632a3b5c70a0e26dd70a2acbe7736e650f602f7"
Jul 29 18:46:59.089352 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:46:59.089326    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=router-gateway-1-openshift-default-56f678f445-9wrf5_kserve-ci-e2e-test(c31372be-e035-44b2-88c2-85e009edb9fc)\"" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5" podUID="c31372be-e035-44b2-88c2-85e009edb9fc"
Jul 29 18:47:02.020832 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:02.020797    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:47:02.021229 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:02.021192    2560 scope.go:117] "RemoveContainer" containerID="dce437b3206937304c61c67ea632a3b5c70a0e26dd70a2acbe7736e650f602f7"
Jul 29 18:47:02.021407 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:47:02.021386    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=router-gateway-1-openshift-default-56f678f445-9wrf5_kserve-ci-e2e-test(c31372be-e035-44b2-88c2-85e009edb9fc)\"" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5" podUID="c31372be-e035-44b2-88c2-85e009edb9fc"
Jul 29 18:47:10.436125 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:10.436066    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:47:16.682703 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:16.682673    2560 scope.go:117] "RemoveContainer" containerID="dce437b3206937304c61c67ea632a3b5c70a0e26dd70a2acbe7736e650f602f7"
Jul 29 18:47:16.683758 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:16.683726    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:47:16.683874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:16.683809    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:47:16.684295 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:16.684249    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:47:17.183520 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:17.183488    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5" event={"ID":"c31372be-e035-44b2-88c2-85e009edb9fc","Type":"ContainerStarted","Data":"a89b06deb98c8f299c92cb6a86eb7e3b1f30e8313200f478d9fa9d288c2a90e8"}
Jul 29 18:47:18.173233 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:18.173192    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:47:18.174311 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:18.174281    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5" podUID="c31372be-e035-44b2-88c2-85e009edb9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 29 18:47:19.173280 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:19.173237    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5" podUID="c31372be-e035-44b2-88c2-85e009edb9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 29 18:47:20.173093 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:20.173047    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5" podUID="c31372be-e035-44b2-88c2-85e009edb9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 29 18:47:21.173583 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:21.173540    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5" podUID="c31372be-e035-44b2-88c2-85e009edb9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 29 18:47:22.173708 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:22.173668    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5" podUID="c31372be-e035-44b2-88c2-85e009edb9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 29 18:47:22.732336 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:22.732299    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"]
Jul 29 18:47:22.732599 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:22.732565    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4" podUID="5335218d-e501-471b-956f-2d578c2e5996" containerName="main" containerID="cri-o://168d266cdf720787e5199ad7af2861b510d1c9fe333806685f1e03b3caf6984b" gracePeriod=30
Jul 29 18:47:22.732796 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:22.732763    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4" podUID="5335218d-e501-471b-956f-2d578c2e5996" containerName="tokenizer" containerID="cri-o://a82e3ef225c4718b177f4981cad795b05ae269774c90be373283ea096e3fbff8" gracePeriod=30
Jul 29 18:47:22.736852 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:22.736791    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:47:22.796012 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:22.795974    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:47:23.177048 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:23.177019    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:47:23.177456 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:23.177329    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:47:23.178059 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:23.178042    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-56f678f445-9wrf5"
Jul 29 18:47:23.226459 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:23.226422    2560 generic.go:356] "Generic (PLEG): container finished" podID="5335218d-e501-471b-956f-2d578c2e5996" containerID="168d266cdf720787e5199ad7af2861b510d1c9fe333806685f1e03b3caf6984b" exitCode=0
Jul 29 18:47:23.226640 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:23.226497    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4" event={"ID":"5335218d-e501-471b-956f-2d578c2e5996","Type":"ContainerDied","Data":"168d266cdf720787e5199ad7af2861b510d1c9fe333806685f1e03b3caf6984b"}
Jul 29 18:47:23.979512 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:23.979485    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:47:24.023996 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.023964    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt2fw\" (UniqueName: \"kubernetes.io/projected/5335218d-e501-471b-956f-2d578c2e5996-kube-api-access-gt2fw\") pod \"5335218d-e501-471b-956f-2d578c2e5996\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") "
Jul 29 18:47:24.024148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.024010    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5335218d-e501-471b-956f-2d578c2e5996-tls-certs\") pod \"5335218d-e501-471b-956f-2d578c2e5996\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") "
Jul 29 18:47:24.024148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.024038    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-cache\") pod \"5335218d-e501-471b-956f-2d578c2e5996\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") "
Jul 29 18:47:24.024148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.024082    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-kserve-provision-location\") pod \"5335218d-e501-471b-956f-2d578c2e5996\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") "
Jul 29 18:47:24.024148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.024129    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-uds\") pod \"5335218d-e501-471b-956f-2d578c2e5996\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") "
Jul 29 18:47:24.024381 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.024170    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-tmp\") pod \"5335218d-e501-471b-956f-2d578c2e5996\" (UID: \"5335218d-e501-471b-956f-2d578c2e5996\") "
Jul 29 18:47:24.024381 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.024324    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "5335218d-e501-471b-956f-2d578c2e5996" (UID: "5335218d-e501-471b-956f-2d578c2e5996"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:47:24.024492 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.024440    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:47:24.024492 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.024437    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "5335218d-e501-471b-956f-2d578c2e5996" (UID: "5335218d-e501-471b-956f-2d578c2e5996"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:47:24.024653 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.024629    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "5335218d-e501-471b-956f-2d578c2e5996" (UID: "5335218d-e501-471b-956f-2d578c2e5996"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:47:24.024892 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.024870    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5335218d-e501-471b-956f-2d578c2e5996" (UID: "5335218d-e501-471b-956f-2d578c2e5996"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:47:24.026263 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.026237    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5335218d-e501-471b-956f-2d578c2e5996-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5335218d-e501-471b-956f-2d578c2e5996" (UID: "5335218d-e501-471b-956f-2d578c2e5996"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:47:24.026381 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.026237    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5335218d-e501-471b-956f-2d578c2e5996-kube-api-access-gt2fw" (OuterVolumeSpecName: "kube-api-access-gt2fw") pod "5335218d-e501-471b-956f-2d578c2e5996" (UID: "5335218d-e501-471b-956f-2d578c2e5996"). InnerVolumeSpecName "kube-api-access-gt2fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:47:24.124802 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.124766    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt2fw\" (UniqueName: \"kubernetes.io/projected/5335218d-e501-471b-956f-2d578c2e5996-kube-api-access-gt2fw\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:47:24.124802 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.124797    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5335218d-e501-471b-956f-2d578c2e5996-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:47:24.124802 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.124808    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:47:24.125034 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.124818    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-uds\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:47:24.125034 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.124827    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5335218d-e501-471b-956f-2d578c2e5996-tokenizer-tmp\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:47:24.235446 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.235408    2560 generic.go:356] "Generic (PLEG): container finished" podID="5335218d-e501-471b-956f-2d578c2e5996" containerID="a82e3ef225c4718b177f4981cad795b05ae269774c90be373283ea096e3fbff8" exitCode=0
Jul 29 18:47:24.235926 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.235473    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"
Jul 29 18:47:24.235926 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.235496    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4" event={"ID":"5335218d-e501-471b-956f-2d578c2e5996","Type":"ContainerDied","Data":"a82e3ef225c4718b177f4981cad795b05ae269774c90be373283ea096e3fbff8"}
Jul 29 18:47:24.235926 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.235602    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4" event={"ID":"5335218d-e501-471b-956f-2d578c2e5996","Type":"ContainerDied","Data":"c584da2a39d8787f5262927e7d15d47a2eeafc4c458794a168804e844edbe132"}
Jul 29 18:47:24.235926 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.235638    2560 scope.go:117] "RemoveContainer" containerID="a82e3ef225c4718b177f4981cad795b05ae269774c90be373283ea096e3fbff8"
Jul 29 18:47:24.245437 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.245419    2560 scope.go:117] "RemoveContainer" containerID="168d266cdf720787e5199ad7af2861b510d1c9fe333806685f1e03b3caf6984b"
Jul 29 18:47:24.253235 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.253192    2560 scope.go:117] "RemoveContainer" containerID="8be22d554457b3e909179d864d057a324d9bdf4aec0f365a964762a1af73a247"
Jul 29 18:47:24.259116 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.259090    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"]
Jul 29 18:47:24.261826 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.261808    2560 scope.go:117] "RemoveContainer" containerID="a82e3ef225c4718b177f4981cad795b05ae269774c90be373283ea096e3fbff8"
Jul 29 18:47:24.262082 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:47:24.262063    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82e3ef225c4718b177f4981cad795b05ae269774c90be373283ea096e3fbff8\": container with ID starting with a82e3ef225c4718b177f4981cad795b05ae269774c90be373283ea096e3fbff8 not found: ID does not exist" containerID="a82e3ef225c4718b177f4981cad795b05ae269774c90be373283ea096e3fbff8"
Jul 29 18:47:24.262146 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.262092    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82e3ef225c4718b177f4981cad795b05ae269774c90be373283ea096e3fbff8"} err="failed to get container status \"a82e3ef225c4718b177f4981cad795b05ae269774c90be373283ea096e3fbff8\": rpc error: code = NotFound desc = could not find container \"a82e3ef225c4718b177f4981cad795b05ae269774c90be373283ea096e3fbff8\": container with ID starting with a82e3ef225c4718b177f4981cad795b05ae269774c90be373283ea096e3fbff8 not found: ID does not exist"
Jul 29 18:47:24.262146 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.262116    2560 scope.go:117] "RemoveContainer" containerID="168d266cdf720787e5199ad7af2861b510d1c9fe333806685f1e03b3caf6984b"
Jul 29 18:47:24.262357 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:47:24.262339    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168d266cdf720787e5199ad7af2861b510d1c9fe333806685f1e03b3caf6984b\": container with ID starting with 168d266cdf720787e5199ad7af2861b510d1c9fe333806685f1e03b3caf6984b not found: ID does not exist" containerID="168d266cdf720787e5199ad7af2861b510d1c9fe333806685f1e03b3caf6984b"
Jul 29 18:47:24.262397 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.262361    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168d266cdf720787e5199ad7af2861b510d1c9fe333806685f1e03b3caf6984b"} err="failed to get container status \"168d266cdf720787e5199ad7af2861b510d1c9fe333806685f1e03b3caf6984b\": rpc error: code = NotFound desc = could not find container \"168d266cdf720787e5199ad7af2861b510d1c9fe333806685f1e03b3caf6984b\": container with ID starting with 168d266cdf720787e5199ad7af2861b510d1c9fe333806685f1e03b3caf6984b not found: ID does not exist"
Jul 29 18:47:24.262397 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.262375    2560 scope.go:117] "RemoveContainer" containerID="8be22d554457b3e909179d864d057a324d9bdf4aec0f365a964762a1af73a247"
Jul 29 18:47:24.262690 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.262667    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-78894kn9f4"]
Jul 29 18:47:24.265928 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:47:24.265896    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be22d554457b3e909179d864d057a324d9bdf4aec0f365a964762a1af73a247\": container with ID starting with 8be22d554457b3e909179d864d057a324d9bdf4aec0f365a964762a1af73a247 not found: ID does not exist" containerID="8be22d554457b3e909179d864d057a324d9bdf4aec0f365a964762a1af73a247"
Jul 29 18:47:24.266025 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.265926    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be22d554457b3e909179d864d057a324d9bdf4aec0f365a964762a1af73a247"} err="failed to get container status \"8be22d554457b3e909179d864d057a324d9bdf4aec0f365a964762a1af73a247\": rpc error: code = NotFound desc = could not find container \"8be22d554457b3e909179d864d057a324d9bdf4aec0f365a964762a1af73a247\": container with ID starting with 8be22d554457b3e909179d864d057a324d9bdf4aec0f365a964762a1af73a247 not found: ID does not exist"
Jul 29 18:47:24.689528 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:24.689486    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5335218d-e501-471b-956f-2d578c2e5996" path="/var/lib/kubelet/pods/5335218d-e501-471b-956f-2d578c2e5996/volumes"
Jul 29 18:47:38.172668 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.172631    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"]
Jul 29 18:47:38.173039 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.172981    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5335218d-e501-471b-956f-2d578c2e5996" containerName="main"
Jul 29 18:47:38.173039 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.172992    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5335218d-e501-471b-956f-2d578c2e5996" containerName="main"
Jul 29 18:47:38.173039 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.173003    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5335218d-e501-471b-956f-2d578c2e5996" containerName="storage-initializer"
Jul 29 18:47:38.173039 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.173009    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5335218d-e501-471b-956f-2d578c2e5996" containerName="storage-initializer"
Jul 29 18:47:38.173039 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.173021    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5335218d-e501-471b-956f-2d578c2e5996" containerName="tokenizer"
Jul 29 18:47:38.173039 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.173026    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5335218d-e501-471b-956f-2d578c2e5996" containerName="tokenizer"
Jul 29 18:47:38.173273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.173085    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5335218d-e501-471b-956f-2d578c2e5996" containerName="tokenizer"
Jul 29 18:47:38.173273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.173093    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5335218d-e501-471b-956f-2d578c2e5996" containerName="main"
Jul 29 18:47:38.176060 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.176040    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.179570 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.179544    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2cmt9\""
Jul 29 18:47:38.179697 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.179604    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 29 18:47:38.185372 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.185344    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"]
Jul 29 18:47:38.201303 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.201272    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:47:38.229119 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.229088    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-home\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.229350 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.229124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-model-cache\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.229350 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.229146    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-dshm\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.229350 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.229163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4caa9b-bbef-4432-be51-1a916be8ab31-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.229350 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.229319    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-758br\" (UniqueName: \"kubernetes.io/projected/5a4caa9b-bbef-4432-be51-1a916be8ab31-kube-api-access-758br\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.229544 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.229366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.229544 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.229393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.330109 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.330072    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-home\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.330109 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.330113    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-model-cache\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.330404 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.330138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-dshm\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.330404 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.330153    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4caa9b-bbef-4432-be51-1a916be8ab31-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.330404 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.330191    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-758br\" (UniqueName: \"kubernetes.io/projected/5a4caa9b-bbef-4432-be51-1a916be8ab31-kube-api-access-758br\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.330404 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.330233    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.330404 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.330248    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.330653 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.330636    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-model-cache\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.330713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.330659    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.330773 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.330748    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.330850 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.330827    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-home\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.332590 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.332570    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-dshm\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.332753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.332735    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4caa9b-bbef-4432-be51-1a916be8ab31-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.338454 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.338431    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-758br\" (UniqueName: \"kubernetes.io/projected/5a4caa9b-bbef-4432-be51-1a916be8ab31-kube-api-access-758br\") pod \"scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.394912 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.394877    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"]
Jul 29 18:47:38.397738 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.397716    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.400421 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.400402    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-7jtvf\""
Jul 29 18:47:38.409413 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.409390    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"]
Jul 29 18:47:38.431187 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.431110    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.431187 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.431168    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6445c25-7e8b-464e-a539-0fe9855a65b6-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.431354 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.431227    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.431354 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.431301    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbdmb\" (UniqueName: \"kubernetes.io/projected/d6445c25-7e8b-464e-a539-0fe9855a65b6-kube-api-access-kbdmb\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.431431 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.431369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.431431 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.431398    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.434597 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.434574    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:47:38.487378 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.487346    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:38.532726 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.532688    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.532872 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.532741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6445c25-7e8b-464e-a539-0fe9855a65b6-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.532872 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.532760    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.532872 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.532808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbdmb\" (UniqueName: \"kubernetes.io/projected/d6445c25-7e8b-464e-a539-0fe9855a65b6-kube-api-access-kbdmb\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.532872 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.532843    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.532872 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.532862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.533164 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.533138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.533256 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.533151    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.533323 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.533251    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.533730 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.533695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.537762 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.536630    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6445c25-7e8b-464e-a539-0fe9855a65b6-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.542164 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.542137    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbdmb\" (UniqueName: \"kubernetes.io/projected/d6445c25-7e8b-464e-a539-0fe9855a65b6-kube-api-access-kbdmb\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.621526 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.621496    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"]
Jul 29 18:47:38.623168 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:47:38.623144    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4caa9b_bbef_4432_be51_1a916be8ab31.slice/crio-378a11e13fa2c6d85165905001710f15a937efc39ecc77ab4925e9782bfaaf42 WatchSource:0}: Error finding container 378a11e13fa2c6d85165905001710f15a937efc39ecc77ab4925e9782bfaaf42: Status 404 returned error can't find the container with id 378a11e13fa2c6d85165905001710f15a937efc39ecc77ab4925e9782bfaaf42
Jul 29 18:47:38.708091 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.708012    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:38.850658 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:38.850624    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"]
Jul 29 18:47:38.852092 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:47:38.852056    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6445c25_7e8b_464e_a539_0fe9855a65b6.slice/crio-5b732d9698b80e085f8c7ab2bf783ebae0de4b9ae2b812ef1c1b1200b6414f3c WatchSource:0}: Error finding container 5b732d9698b80e085f8c7ab2bf783ebae0de4b9ae2b812ef1c1b1200b6414f3c: Status 404 returned error can't find the container with id 5b732d9698b80e085f8c7ab2bf783ebae0de4b9ae2b812ef1c1b1200b6414f3c
Jul 29 18:47:39.318111 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:39.318071    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx" event={"ID":"5a4caa9b-bbef-4432-be51-1a916be8ab31","Type":"ContainerStarted","Data":"2f149ba15be4b0fd75252737212f00ea1b24ed00d6e1ee7818e06735d3025df4"}
Jul 29 18:47:39.318554 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:39.318136    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx" event={"ID":"5a4caa9b-bbef-4432-be51-1a916be8ab31","Type":"ContainerStarted","Data":"378a11e13fa2c6d85165905001710f15a937efc39ecc77ab4925e9782bfaaf42"}
Jul 29 18:47:39.319965 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:39.319939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr" event={"ID":"d6445c25-7e8b-464e-a539-0fe9855a65b6","Type":"ContainerStarted","Data":"b92afcfc89e80a81b7cd75336df04e4a04b4887d97da64f2603fa9df05a159b3"}
Jul 29 18:47:39.320074 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:39.319990    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr" event={"ID":"d6445c25-7e8b-464e-a539-0fe9855a65b6","Type":"ContainerStarted","Data":"5b732d9698b80e085f8c7ab2bf783ebae0de4b9ae2b812ef1c1b1200b6414f3c"}
Jul 29 18:47:40.327351 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:40.327311    2560 generic.go:356] "Generic (PLEG): container finished" podID="d6445c25-7e8b-464e-a539-0fe9855a65b6" containerID="b92afcfc89e80a81b7cd75336df04e4a04b4887d97da64f2603fa9df05a159b3" exitCode=0
Jul 29 18:47:40.327877 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:40.327356    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr" event={"ID":"d6445c25-7e8b-464e-a539-0fe9855a65b6","Type":"ContainerDied","Data":"b92afcfc89e80a81b7cd75336df04e4a04b4887d97da64f2603fa9df05a159b3"}
Jul 29 18:47:41.334923 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:41.334885    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr" event={"ID":"d6445c25-7e8b-464e-a539-0fe9855a65b6","Type":"ContainerStarted","Data":"4ee88c15afb2df3fb0bfec6075a86fbb27f824b50898c2dcfb2cc267eb37e6d6"}
Jul 29 18:47:41.335391 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:41.334951    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:41.335391 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:41.334975    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr" event={"ID":"d6445c25-7e8b-464e-a539-0fe9855a65b6","Type":"ContainerStarted","Data":"6a320b1353296c3e65082b461d31aecf5548f44182be5d9f954c9f17a845b5ed"}
Jul 29 18:47:41.358838 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:41.358779    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr" podStartSLOduration=2.607092448 podStartE2EDuration="3.35876347s" totalImagesPullingTime="0s" totalInitContainerRuntime="751.671022ms" isStatefulPod=false podCreationTimestamp="2026-07-29 18:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:47:41.357447155 +0000 UTC m=+787.251447558" watchObservedRunningTime="2026-07-29 18:47:41.35876347 +0000 UTC m=+787.252763871"
Jul 29 18:47:46.368551 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:46.368514    2560 generic.go:356] "Generic (PLEG): container finished" podID="5a4caa9b-bbef-4432-be51-1a916be8ab31" containerID="2f149ba15be4b0fd75252737212f00ea1b24ed00d6e1ee7818e06735d3025df4" exitCode=0
Jul 29 18:47:46.368991 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:46.368588    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx" event={"ID":"5a4caa9b-bbef-4432-be51-1a916be8ab31","Type":"ContainerDied","Data":"2f149ba15be4b0fd75252737212f00ea1b24ed00d6e1ee7818e06735d3025df4"}
Jul 29 18:47:48.381917 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:48.381882    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx" event={"ID":"5a4caa9b-bbef-4432-be51-1a916be8ab31","Type":"ContainerStarted","Data":"4152af5ba4030bd21a6f51ed3170fc0a8bbf1232629b5afced3c1be1fbda0968"}
Jul 29 18:47:48.401949 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:48.401899    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx" podStartSLOduration=2.075823481 podStartE2EDuration="10.401880567s" totalImagesPullingTime="1.063638808s" totalInitContainerRuntime="7.262418278s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:47:48.400477821 +0000 UTC m=+794.294478244" watchObservedRunningTime="2026-07-29 18:47:48.401880567 +0000 UTC m=+794.295880967"
Jul 29 18:47:48.488164 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:48.488126    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:48.488364 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:48.488225    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:48.503496 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:48.503462    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:48.708786 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:48.708697    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:48.708786 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:48.708760    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:48.711939 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:48.711917    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:49.388737 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:49.388712    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:47:49.398821 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:49.398797    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:47:50.317148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:47:50.317108    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:48:10.394625 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:10.394542    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:48:11.493595 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:11.493561    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:48:11.784209 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:11.784090    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"]
Jul 29 18:48:11.784452 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:11.784417    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr" podUID="d6445c25-7e8b-464e-a539-0fe9855a65b6" containerName="main" containerID="cri-o://6a320b1353296c3e65082b461d31aecf5548f44182be5d9f954c9f17a845b5ed" gracePeriod=30
Jul 29 18:48:11.784865 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:11.784688    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr" podUID="d6445c25-7e8b-464e-a539-0fe9855a65b6" containerName="tokenizer" containerID="cri-o://4ee88c15afb2df3fb0bfec6075a86fbb27f824b50898c2dcfb2cc267eb37e6d6" gracePeriod=30
Jul 29 18:48:11.788156 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:11.788130    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"]
Jul 29 18:48:11.788731 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:11.788682    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx" podUID="5a4caa9b-bbef-4432-be51-1a916be8ab31" containerName="main" containerID="cri-o://4152af5ba4030bd21a6f51ed3170fc0a8bbf1232629b5afced3c1be1fbda0968" gracePeriod=30
Jul 29 18:48:11.790461 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:11.790437    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:48:11.855495 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:11.855461    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:48:12.052608 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.052582    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:48:12.239343 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.239298    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-dshm\") pod \"5a4caa9b-bbef-4432-be51-1a916be8ab31\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") "
Jul 29 18:48:12.239525 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.239356    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-tmp-dir\") pod \"5a4caa9b-bbef-4432-be51-1a916be8ab31\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") "
Jul 29 18:48:12.239525 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.239437    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4caa9b-bbef-4432-be51-1a916be8ab31-tls-certs\") pod \"5a4caa9b-bbef-4432-be51-1a916be8ab31\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") "
Jul 29 18:48:12.239525 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.239474    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-kserve-provision-location\") pod \"5a4caa9b-bbef-4432-be51-1a916be8ab31\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") "
Jul 29 18:48:12.239682 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.239534    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-home\") pod \"5a4caa9b-bbef-4432-be51-1a916be8ab31\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") "
Jul 29 18:48:12.239682 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.239561    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-758br\" (UniqueName: \"kubernetes.io/projected/5a4caa9b-bbef-4432-be51-1a916be8ab31-kube-api-access-758br\") pod \"5a4caa9b-bbef-4432-be51-1a916be8ab31\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") "
Jul 29 18:48:12.239682 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.239598    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-model-cache\") pod \"5a4caa9b-bbef-4432-be51-1a916be8ab31\" (UID: \"5a4caa9b-bbef-4432-be51-1a916be8ab31\") "
Jul 29 18:48:12.239682 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.239643    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a4caa9b-bbef-4432-be51-1a916be8ab31" (UID: "5a4caa9b-bbef-4432-be51-1a916be8ab31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:48:12.239901 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.239800    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-home" (OuterVolumeSpecName: "home") pod "5a4caa9b-bbef-4432-be51-1a916be8ab31" (UID: "5a4caa9b-bbef-4432-be51-1a916be8ab31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:48:12.239901 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.239875    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:12.239901 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.239892    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:12.240066 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.239897    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a4caa9b-bbef-4432-be51-1a916be8ab31" (UID: "5a4caa9b-bbef-4432-be51-1a916be8ab31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:48:12.241865 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.241826    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4caa9b-bbef-4432-be51-1a916be8ab31-kube-api-access-758br" (OuterVolumeSpecName: "kube-api-access-758br") pod "5a4caa9b-bbef-4432-be51-1a916be8ab31" (UID: "5a4caa9b-bbef-4432-be51-1a916be8ab31"). InnerVolumeSpecName "kube-api-access-758br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:48:12.241996 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.241906    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4caa9b-bbef-4432-be51-1a916be8ab31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a4caa9b-bbef-4432-be51-1a916be8ab31" (UID: "5a4caa9b-bbef-4432-be51-1a916be8ab31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:48:12.241996 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.241925    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-dshm" (OuterVolumeSpecName: "dshm") pod "5a4caa9b-bbef-4432-be51-1a916be8ab31" (UID: "5a4caa9b-bbef-4432-be51-1a916be8ab31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:48:12.295406 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.295310    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a4caa9b-bbef-4432-be51-1a916be8ab31" (UID: "5a4caa9b-bbef-4432-be51-1a916be8ab31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:48:12.340724 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.340687    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-758br\" (UniqueName: \"kubernetes.io/projected/5a4caa9b-bbef-4432-be51-1a916be8ab31-kube-api-access-758br\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:12.340724 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.340716    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:12.340724 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.340726    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:12.340724 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.340734    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4caa9b-bbef-4432-be51-1a916be8ab31-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:12.340996 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.340743    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4caa9b-bbef-4432-be51-1a916be8ab31-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:12.506885 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.506851    2560 generic.go:356] "Generic (PLEG): container finished" podID="5a4caa9b-bbef-4432-be51-1a916be8ab31" containerID="4152af5ba4030bd21a6f51ed3170fc0a8bbf1232629b5afced3c1be1fbda0968" exitCode=0
Jul 29 18:48:12.507400 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.506923    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"
Jul 29 18:48:12.507400 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.506945    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx" event={"ID":"5a4caa9b-bbef-4432-be51-1a916be8ab31","Type":"ContainerDied","Data":"4152af5ba4030bd21a6f51ed3170fc0a8bbf1232629b5afced3c1be1fbda0968"}
Jul 29 18:48:12.507400 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.507007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx" event={"ID":"5a4caa9b-bbef-4432-be51-1a916be8ab31","Type":"ContainerDied","Data":"378a11e13fa2c6d85165905001710f15a937efc39ecc77ab4925e9782bfaaf42"}
Jul 29 18:48:12.507400 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.507037    2560 scope.go:117] "RemoveContainer" containerID="4152af5ba4030bd21a6f51ed3170fc0a8bbf1232629b5afced3c1be1fbda0968"
Jul 29 18:48:12.509469 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.509447    2560 generic.go:356] "Generic (PLEG): container finished" podID="d6445c25-7e8b-464e-a539-0fe9855a65b6" containerID="6a320b1353296c3e65082b461d31aecf5548f44182be5d9f954c9f17a845b5ed" exitCode=0
Jul 29 18:48:12.509607 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.509488    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr" event={"ID":"d6445c25-7e8b-464e-a539-0fe9855a65b6","Type":"ContainerDied","Data":"6a320b1353296c3e65082b461d31aecf5548f44182be5d9f954c9f17a845b5ed"}
Jul 29 18:48:12.517051 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.517034    2560 scope.go:117] "RemoveContainer" containerID="2f149ba15be4b0fd75252737212f00ea1b24ed00d6e1ee7818e06735d3025df4"
Jul 29 18:48:12.532133 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.532097    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"]
Jul 29 18:48:12.536179 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.536153    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-66878cbf8-m4dmx"]
Jul 29 18:48:12.580918 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.580896    2560 scope.go:117] "RemoveContainer" containerID="4152af5ba4030bd21a6f51ed3170fc0a8bbf1232629b5afced3c1be1fbda0968"
Jul 29 18:48:12.581298 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:48:12.581279    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4152af5ba4030bd21a6f51ed3170fc0a8bbf1232629b5afced3c1be1fbda0968\": container with ID starting with 4152af5ba4030bd21a6f51ed3170fc0a8bbf1232629b5afced3c1be1fbda0968 not found: ID does not exist" containerID="4152af5ba4030bd21a6f51ed3170fc0a8bbf1232629b5afced3c1be1fbda0968"
Jul 29 18:48:12.581360 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.581309    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4152af5ba4030bd21a6f51ed3170fc0a8bbf1232629b5afced3c1be1fbda0968"} err="failed to get container status \"4152af5ba4030bd21a6f51ed3170fc0a8bbf1232629b5afced3c1be1fbda0968\": rpc error: code = NotFound desc = could not find container \"4152af5ba4030bd21a6f51ed3170fc0a8bbf1232629b5afced3c1be1fbda0968\": container with ID starting with 4152af5ba4030bd21a6f51ed3170fc0a8bbf1232629b5afced3c1be1fbda0968 not found: ID does not exist"
Jul 29 18:48:12.581360 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.581330    2560 scope.go:117] "RemoveContainer" containerID="2f149ba15be4b0fd75252737212f00ea1b24ed00d6e1ee7818e06735d3025df4"
Jul 29 18:48:12.581634 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:48:12.581609    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f149ba15be4b0fd75252737212f00ea1b24ed00d6e1ee7818e06735d3025df4\": container with ID starting with 2f149ba15be4b0fd75252737212f00ea1b24ed00d6e1ee7818e06735d3025df4 not found: ID does not exist" containerID="2f149ba15be4b0fd75252737212f00ea1b24ed00d6e1ee7818e06735d3025df4"
Jul 29 18:48:12.581731 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.581645    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f149ba15be4b0fd75252737212f00ea1b24ed00d6e1ee7818e06735d3025df4"} err="failed to get container status \"2f149ba15be4b0fd75252737212f00ea1b24ed00d6e1ee7818e06735d3025df4\": rpc error: code = NotFound desc = could not find container \"2f149ba15be4b0fd75252737212f00ea1b24ed00d6e1ee7818e06735d3025df4\": container with ID starting with 2f149ba15be4b0fd75252737212f00ea1b24ed00d6e1ee7818e06735d3025df4 not found: ID does not exist"
Jul 29 18:48:12.687684 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:12.687649    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a4caa9b-bbef-4432-be51-1a916be8ab31" path="/var/lib/kubelet/pods/5a4caa9b-bbef-4432-be51-1a916be8ab31/volumes"
Jul 29 18:48:13.024002 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.023980    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:48:13.045102 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.045081    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-tmp\") pod \"d6445c25-7e8b-464e-a539-0fe9855a65b6\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") "
Jul 29 18:48:13.045174 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.045113    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbdmb\" (UniqueName: \"kubernetes.io/projected/d6445c25-7e8b-464e-a539-0fe9855a65b6-kube-api-access-kbdmb\") pod \"d6445c25-7e8b-464e-a539-0fe9855a65b6\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") "
Jul 29 18:48:13.045174 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.045134    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6445c25-7e8b-464e-a539-0fe9855a65b6-tls-certs\") pod \"d6445c25-7e8b-464e-a539-0fe9855a65b6\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") "
Jul 29 18:48:13.045347 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.045322    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-cache\") pod \"d6445c25-7e8b-464e-a539-0fe9855a65b6\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") "
Jul 29 18:48:13.045389 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.045381    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-uds\") pod \"d6445c25-7e8b-464e-a539-0fe9855a65b6\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") "
Jul 29 18:48:13.045452 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.045434    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-kserve-provision-location\") pod \"d6445c25-7e8b-464e-a539-0fe9855a65b6\" (UID: \"d6445c25-7e8b-464e-a539-0fe9855a65b6\") "
Jul 29 18:48:13.045452 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.045438    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "d6445c25-7e8b-464e-a539-0fe9855a65b6" (UID: "d6445c25-7e8b-464e-a539-0fe9855a65b6"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:48:13.045600 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.045575    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "d6445c25-7e8b-464e-a539-0fe9855a65b6" (UID: "d6445c25-7e8b-464e-a539-0fe9855a65b6"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:48:13.045600 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.045588    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d6445c25-7e8b-464e-a539-0fe9855a65b6" (UID: "d6445c25-7e8b-464e-a539-0fe9855a65b6"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:48:13.045787 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.045766    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:13.045858 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.045791    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-uds\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:13.045858 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.045806    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-tokenizer-tmp\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:13.046191 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.046158    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6445c25-7e8b-464e-a539-0fe9855a65b6" (UID: "d6445c25-7e8b-464e-a539-0fe9855a65b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:48:13.047539 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.047516    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6445c25-7e8b-464e-a539-0fe9855a65b6-kube-api-access-kbdmb" (OuterVolumeSpecName: "kube-api-access-kbdmb") pod "d6445c25-7e8b-464e-a539-0fe9855a65b6" (UID: "d6445c25-7e8b-464e-a539-0fe9855a65b6"). InnerVolumeSpecName "kube-api-access-kbdmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:48:13.047744 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.047726    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6445c25-7e8b-464e-a539-0fe9855a65b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6445c25-7e8b-464e-a539-0fe9855a65b6" (UID: "d6445c25-7e8b-464e-a539-0fe9855a65b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:48:13.146538 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.146455    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbdmb\" (UniqueName: \"kubernetes.io/projected/d6445c25-7e8b-464e-a539-0fe9855a65b6-kube-api-access-kbdmb\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:13.146538 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.146484    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6445c25-7e8b-464e-a539-0fe9855a65b6-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:13.146538 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.146496    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6445c25-7e8b-464e-a539-0fe9855a65b6-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:13.516114 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.516025    2560 generic.go:356] "Generic (PLEG): container finished" podID="d6445c25-7e8b-464e-a539-0fe9855a65b6" containerID="4ee88c15afb2df3fb0bfec6075a86fbb27f824b50898c2dcfb2cc267eb37e6d6" exitCode=0
Jul 29 18:48:13.516114 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.516084    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"
Jul 29 18:48:13.516586 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.516079    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr" event={"ID":"d6445c25-7e8b-464e-a539-0fe9855a65b6","Type":"ContainerDied","Data":"4ee88c15afb2df3fb0bfec6075a86fbb27f824b50898c2dcfb2cc267eb37e6d6"}
Jul 29 18:48:13.516586 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.516228    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr" event={"ID":"d6445c25-7e8b-464e-a539-0fe9855a65b6","Type":"ContainerDied","Data":"5b732d9698b80e085f8c7ab2bf783ebae0de4b9ae2b812ef1c1b1200b6414f3c"}
Jul 29 18:48:13.516586 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.516264    2560 scope.go:117] "RemoveContainer" containerID="4ee88c15afb2df3fb0bfec6075a86fbb27f824b50898c2dcfb2cc267eb37e6d6"
Jul 29 18:48:13.525345 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.525325    2560 scope.go:117] "RemoveContainer" containerID="6a320b1353296c3e65082b461d31aecf5548f44182be5d9f954c9f17a845b5ed"
Jul 29 18:48:13.533007 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.532989    2560 scope.go:117] "RemoveContainer" containerID="b92afcfc89e80a81b7cd75336df04e4a04b4887d97da64f2603fa9df05a159b3"
Jul 29 18:48:13.541354 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.541190    2560 scope.go:117] "RemoveContainer" containerID="4ee88c15afb2df3fb0bfec6075a86fbb27f824b50898c2dcfb2cc267eb37e6d6"
Jul 29 18:48:13.541731 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:48:13.541707    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee88c15afb2df3fb0bfec6075a86fbb27f824b50898c2dcfb2cc267eb37e6d6\": container with ID starting with 4ee88c15afb2df3fb0bfec6075a86fbb27f824b50898c2dcfb2cc267eb37e6d6 not found: ID does not exist" containerID="4ee88c15afb2df3fb0bfec6075a86fbb27f824b50898c2dcfb2cc267eb37e6d6"
Jul 29 18:48:13.541801 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.541739    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee88c15afb2df3fb0bfec6075a86fbb27f824b50898c2dcfb2cc267eb37e6d6"} err="failed to get container status \"4ee88c15afb2df3fb0bfec6075a86fbb27f824b50898c2dcfb2cc267eb37e6d6\": rpc error: code = NotFound desc = could not find container \"4ee88c15afb2df3fb0bfec6075a86fbb27f824b50898c2dcfb2cc267eb37e6d6\": container with ID starting with 4ee88c15afb2df3fb0bfec6075a86fbb27f824b50898c2dcfb2cc267eb37e6d6 not found: ID does not exist"
Jul 29 18:48:13.541801 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.541757    2560 scope.go:117] "RemoveContainer" containerID="6a320b1353296c3e65082b461d31aecf5548f44182be5d9f954c9f17a845b5ed"
Jul 29 18:48:13.542024 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:48:13.542006    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a320b1353296c3e65082b461d31aecf5548f44182be5d9f954c9f17a845b5ed\": container with ID starting with 6a320b1353296c3e65082b461d31aecf5548f44182be5d9f954c9f17a845b5ed not found: ID does not exist" containerID="6a320b1353296c3e65082b461d31aecf5548f44182be5d9f954c9f17a845b5ed"
Jul 29 18:48:13.542090 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.542030    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a320b1353296c3e65082b461d31aecf5548f44182be5d9f954c9f17a845b5ed"} err="failed to get container status \"6a320b1353296c3e65082b461d31aecf5548f44182be5d9f954c9f17a845b5ed\": rpc error: code = NotFound desc = could not find container \"6a320b1353296c3e65082b461d31aecf5548f44182be5d9f954c9f17a845b5ed\": container with ID starting with 6a320b1353296c3e65082b461d31aecf5548f44182be5d9f954c9f17a845b5ed not found: ID does not exist"
Jul 29 18:48:13.542090 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.542045    2560 scope.go:117] "RemoveContainer" containerID="b92afcfc89e80a81b7cd75336df04e4a04b4887d97da64f2603fa9df05a159b3"
Jul 29 18:48:13.542329 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:48:13.542311    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b92afcfc89e80a81b7cd75336df04e4a04b4887d97da64f2603fa9df05a159b3\": container with ID starting with b92afcfc89e80a81b7cd75336df04e4a04b4887d97da64f2603fa9df05a159b3 not found: ID does not exist" containerID="b92afcfc89e80a81b7cd75336df04e4a04b4887d97da64f2603fa9df05a159b3"
Jul 29 18:48:13.542379 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.542334    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b92afcfc89e80a81b7cd75336df04e4a04b4887d97da64f2603fa9df05a159b3"} err="failed to get container status \"b92afcfc89e80a81b7cd75336df04e4a04b4887d97da64f2603fa9df05a159b3\": rpc error: code = NotFound desc = could not find container \"b92afcfc89e80a81b7cd75336df04e4a04b4887d97da64f2603fa9df05a159b3\": container with ID starting with b92afcfc89e80a81b7cd75336df04e4a04b4887d97da64f2603fa9df05a159b3 not found: ID does not exist"
Jul 29 18:48:13.543420 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.543398    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"]
Jul 29 18:48:13.549389 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:13.549369    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6b9c4j22pr"]
Jul 29 18:48:14.687176 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:14.687138    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6445c25-7e8b-464e-a539-0fe9855a65b6" path="/var/lib/kubelet/pods/d6445c25-7e8b-464e-a539-0fe9855a65b6/volumes"
Jul 29 18:48:25.371101 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.371060    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"]
Jul 29 18:48:25.371563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.371437    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4caa9b-bbef-4432-be51-1a916be8ab31" containerName="storage-initializer"
Jul 29 18:48:25.371563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.371451    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4caa9b-bbef-4432-be51-1a916be8ab31" containerName="storage-initializer"
Jul 29 18:48:25.371563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.371461    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6445c25-7e8b-464e-a539-0fe9855a65b6" containerName="storage-initializer"
Jul 29 18:48:25.371563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.371466    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6445c25-7e8b-464e-a539-0fe9855a65b6" containerName="storage-initializer"
Jul 29 18:48:25.371563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.371479    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6445c25-7e8b-464e-a539-0fe9855a65b6" containerName="tokenizer"
Jul 29 18:48:25.371563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.371487    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6445c25-7e8b-464e-a539-0fe9855a65b6" containerName="tokenizer"
Jul 29 18:48:25.371563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.371493    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4caa9b-bbef-4432-be51-1a916be8ab31" containerName="main"
Jul 29 18:48:25.371563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.371498    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4caa9b-bbef-4432-be51-1a916be8ab31" containerName="main"
Jul 29 18:48:25.371563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.371506    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6445c25-7e8b-464e-a539-0fe9855a65b6" containerName="main"
Jul 29 18:48:25.371563 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.371510    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6445c25-7e8b-464e-a539-0fe9855a65b6" containerName="main"
Jul 29 18:48:25.371876 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.371574    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6445c25-7e8b-464e-a539-0fe9855a65b6" containerName="main"
Jul 29 18:48:25.371876 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.371581    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6445c25-7e8b-464e-a539-0fe9855a65b6" containerName="tokenizer"
Jul 29 18:48:25.371876 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.371588    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a4caa9b-bbef-4432-be51-1a916be8ab31" containerName="main"
Jul 29 18:48:25.374889 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.374869    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.378399 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.378372    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2cmt9\""
Jul 29 18:48:25.378512 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.378386    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 29 18:48:25.387090 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.387065    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"]
Jul 29 18:48:25.393268 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.393242    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:48:25.431708 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.431675    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.431879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.431725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-dshm\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.431879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.431746    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.431879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.431773    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-home\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.431879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.431792    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-model-cache\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.431879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.431857    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.431879 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.431874    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpm9v\" (UniqueName: \"kubernetes.io/projected/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-kube-api-access-lpm9v\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.532667 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.532621    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-dshm\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.532946 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.532673    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.532946 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.532704    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-home\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.532946 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.532732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-model-cache\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.532946 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.532819    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.532946 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.532843    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpm9v\" (UniqueName: \"kubernetes.io/projected/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-kube-api-access-lpm9v\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.532946 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.532891    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.533327 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.533227    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-model-cache\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.533327 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.533286    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.533529 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.533499    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.533748 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.533726    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-home\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.535597 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.535571    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.535688 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.535618    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-dshm\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.542746 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.542722    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpm9v\" (UniqueName: \"kubernetes.io/projected/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-kube-api-access-lpm9v\") pod \"scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.609744 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.609713    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"]
Jul 29 18:48:25.612615 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.612594    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.615058 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.615037    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-rd7g5\""
Jul 29 18:48:25.630626 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.630567    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"]
Jul 29 18:48:25.633594 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.633562    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.633729 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.633655    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8jv\" (UniqueName: \"kubernetes.io/projected/de5c5069-8943-424d-a642-73377008824e-kube-api-access-jq8jv\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.633804 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.633736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.633804 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.633769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5c5069-8943-424d-a642-73377008824e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.633804 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.633799    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.633974 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.633841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.645581 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.645541    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:48:25.686524 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.686483    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:25.734967 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.734916    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5c5069-8943-424d-a642-73377008824e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.735130 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.735003    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.735130 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.735037    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.735130 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.735077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.735298 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.735138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8jv\" (UniqueName: \"kubernetes.io/projected/de5c5069-8943-424d-a642-73377008824e-kube-api-access-jq8jv\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.735298 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.735221    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.735490 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.735465    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.735569 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.735505    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.735569 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.735530    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.735645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.735579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.737643 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.737617    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5c5069-8943-424d-a642-73377008824e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.745633 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.745165    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8jv\" (UniqueName: \"kubernetes.io/projected/de5c5069-8943-424d-a642-73377008824e-kube-api-access-jq8jv\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:25.832528 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.832498    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"]
Jul 29 18:48:25.834386 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:48:25.834357    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733a9f9b_8e2d_45e7_8ecb_3d3c1b7c2aa4.slice/crio-9ca641de27e1f3918798d4f13593ce8db4f9e364978cf8d336c0f851466122f6 WatchSource:0}: Error finding container 9ca641de27e1f3918798d4f13593ce8db4f9e364978cf8d336c0f851466122f6: Status 404 returned error can't find the container with id 9ca641de27e1f3918798d4f13593ce8db4f9e364978cf8d336c0f851466122f6
Jul 29 18:48:25.922757 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:25.922728    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:26.066632 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:26.066606    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"]
Jul 29 18:48:26.068501 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:48:26.068465    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5c5069_8943_424d_a642_73377008824e.slice/crio-df0196fa2342744c2892266a6f3c28fe40e151c43acff0bc7af79c9f8ca6855d WatchSource:0}: Error finding container df0196fa2342744c2892266a6f3c28fe40e151c43acff0bc7af79c9f8ca6855d: Status 404 returned error can't find the container with id df0196fa2342744c2892266a6f3c28fe40e151c43acff0bc7af79c9f8ca6855d
Jul 29 18:48:26.585370 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:26.585323    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m" event={"ID":"de5c5069-8943-424d-a642-73377008824e","Type":"ContainerStarted","Data":"5f82308ae468c87caa72fee5e923f2cbea16c8528b28b3abff51185e59eb8f8c"}
Jul 29 18:48:26.585810 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:26.585436    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m" event={"ID":"de5c5069-8943-424d-a642-73377008824e","Type":"ContainerStarted","Data":"df0196fa2342744c2892266a6f3c28fe40e151c43acff0bc7af79c9f8ca6855d"}
Jul 29 18:48:26.587042 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:26.587015    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6" event={"ID":"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4","Type":"ContainerStarted","Data":"a30f113662381e1b266a68d841968bd108376100eba59ddf2f45504a17b222dc"}
Jul 29 18:48:26.587150 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:26.587059    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6" event={"ID":"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4","Type":"ContainerStarted","Data":"9ca641de27e1f3918798d4f13593ce8db4f9e364978cf8d336c0f851466122f6"}
Jul 29 18:48:27.594792 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:27.594742    2560 generic.go:356] "Generic (PLEG): container finished" podID="de5c5069-8943-424d-a642-73377008824e" containerID="5f82308ae468c87caa72fee5e923f2cbea16c8528b28b3abff51185e59eb8f8c" exitCode=0
Jul 29 18:48:27.595218 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:27.594867    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m" event={"ID":"de5c5069-8943-424d-a642-73377008824e","Type":"ContainerDied","Data":"5f82308ae468c87caa72fee5e923f2cbea16c8528b28b3abff51185e59eb8f8c"}
Jul 29 18:48:28.602381 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:28.602338    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m" event={"ID":"de5c5069-8943-424d-a642-73377008824e","Type":"ContainerStarted","Data":"7d7e93c669c0e313d0a587d2d2c2775ffe4535e1a67920af466ec9ef165542e2"}
Jul 29 18:48:28.602755 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:28.602428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m" event={"ID":"de5c5069-8943-424d-a642-73377008824e","Type":"ContainerStarted","Data":"54a98a1d209d7b7fa977b99a9fbd9b9088da509a45c4b474505fbf0a0de2cb7f"}
Jul 29 18:48:28.602755 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:28.602592    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:28.623898 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:28.623846    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m" podStartSLOduration=2.859335326 podStartE2EDuration="3.623831225s" totalImagesPullingTime="0s" totalInitContainerRuntime="764.495899ms" isStatefulPod=false podCreationTimestamp="2026-07-29 18:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:48:28.622810614 +0000 UTC m=+834.516811036" watchObservedRunningTime="2026-07-29 18:48:28.623831225 +0000 UTC m=+834.517831626"
Jul 29 18:48:31.625247 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:31.625132    2560 generic.go:356] "Generic (PLEG): container finished" podID="733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" containerID="a30f113662381e1b266a68d841968bd108376100eba59ddf2f45504a17b222dc" exitCode=0
Jul 29 18:48:31.625247 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:31.625221    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6" event={"ID":"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4","Type":"ContainerDied","Data":"a30f113662381e1b266a68d841968bd108376100eba59ddf2f45504a17b222dc"}
Jul 29 18:48:32.633066 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:32.633025    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6" event={"ID":"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4","Type":"ContainerStarted","Data":"d2f3a69ac002f709642d6b5acc51e6ce32685ff2709d7ffe6ea22241699171fe"}
Jul 29 18:48:32.655326 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:32.655277    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6" podStartSLOduration=2.191702987 podStartE2EDuration="7.655260575s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.463557588s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:48:32.65296517 +0000 UTC m=+838.546965598" watchObservedRunningTime="2026-07-29 18:48:32.655260575 +0000 UTC m=+838.549260977"
Jul 29 18:48:35.687281 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:35.687249    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:35.687671 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:35.687295    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:35.700461 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:35.700437    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:35.922862 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:35.922820    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:35.923222 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:35.923174    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:35.926038 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:35.926008    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:36.661749 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:36.661719    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:36.672015 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:36.671988    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:37.060189 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:37.060157    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:48:57.667520 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:57.667489    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:48:58.551574 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:58.551538    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:48:58.608899 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:58.608868    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:48:59.298816 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.298770    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"]
Jul 29 18:48:59.299257 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.298999    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m" podUID="de5c5069-8943-424d-a642-73377008824e" containerName="main" containerID="cri-o://54a98a1d209d7b7fa977b99a9fbd9b9088da509a45c4b474505fbf0a0de2cb7f" gracePeriod=30
Jul 29 18:48:59.299257 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.299046    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m" podUID="de5c5069-8943-424d-a642-73377008824e" containerName="tokenizer" containerID="cri-o://7d7e93c669c0e313d0a587d2d2c2775ffe4535e1a67920af466ec9ef165542e2" gracePeriod=30
Jul 29 18:48:59.303393 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.303363    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:48:59.305864 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.305835    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"]
Jul 29 18:48:59.306135 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.306084    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6" podUID="733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" containerName="main" containerID="cri-o://d2f3a69ac002f709642d6b5acc51e6ce32685ff2709d7ffe6ea22241699171fe" gracePeriod=30
Jul 29 18:48:59.398947 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.398911    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:48:59.567207 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.567165    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:59.739998 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.739960    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpm9v\" (UniqueName: \"kubernetes.io/projected/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-kube-api-access-lpm9v\") pod \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") "
Jul 29 18:48:59.740227 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.740018    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-tls-certs\") pod \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") "
Jul 29 18:48:59.740227 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.740051    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-dshm\") pod \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") "
Jul 29 18:48:59.740227 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.740072    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-home\") pod \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") "
Jul 29 18:48:59.740227 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.740105    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-model-cache\") pod \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") "
Jul 29 18:48:59.740227 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.740122    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-tmp-dir\") pod \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") "
Jul 29 18:48:59.740227 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.740142    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-kserve-provision-location\") pod \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\" (UID: \"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4\") "
Jul 29 18:48:59.740567 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.740378    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-model-cache" (OuterVolumeSpecName: "model-cache") pod "733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" (UID: "733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:48:59.740567 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.740398    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-home" (OuterVolumeSpecName: "home") pod "733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" (UID: "733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:48:59.740567 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.740541    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" (UID: "733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:48:59.742266 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.742236    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-dshm" (OuterVolumeSpecName: "dshm") pod "733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" (UID: "733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:48:59.742369 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.742332    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-kube-api-access-lpm9v" (OuterVolumeSpecName: "kube-api-access-lpm9v") pod "733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" (UID: "733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4"). InnerVolumeSpecName "kube-api-access-lpm9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:48:59.742479 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.742457    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" (UID: "733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:48:59.782108 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.782077    2560 generic.go:356] "Generic (PLEG): container finished" podID="de5c5069-8943-424d-a642-73377008824e" containerID="54a98a1d209d7b7fa977b99a9fbd9b9088da509a45c4b474505fbf0a0de2cb7f" exitCode=0
Jul 29 18:48:59.782273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.782154    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m" event={"ID":"de5c5069-8943-424d-a642-73377008824e","Type":"ContainerDied","Data":"54a98a1d209d7b7fa977b99a9fbd9b9088da509a45c4b474505fbf0a0de2cb7f"}
Jul 29 18:48:59.783793 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.783767    2560 generic.go:356] "Generic (PLEG): container finished" podID="733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" containerID="d2f3a69ac002f709642d6b5acc51e6ce32685ff2709d7ffe6ea22241699171fe" exitCode=0
Jul 29 18:48:59.783931 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.783830    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"
Jul 29 18:48:59.783931 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.783847    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6" event={"ID":"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4","Type":"ContainerDied","Data":"d2f3a69ac002f709642d6b5acc51e6ce32685ff2709d7ffe6ea22241699171fe"}
Jul 29 18:48:59.783931 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.783895    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6" event={"ID":"733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4","Type":"ContainerDied","Data":"9ca641de27e1f3918798d4f13593ce8db4f9e364978cf8d336c0f851466122f6"}
Jul 29 18:48:59.783931 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.783927    2560 scope.go:117] "RemoveContainer" containerID="d2f3a69ac002f709642d6b5acc51e6ce32685ff2709d7ffe6ea22241699171fe"
Jul 29 18:48:59.800445 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.800413    2560 scope.go:117] "RemoveContainer" containerID="a30f113662381e1b266a68d841968bd108376100eba59ddf2f45504a17b222dc"
Jul 29 18:48:59.802852 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.802770    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" (UID: "733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:48:59.841015 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.840967    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:59.841015 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.841014    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpm9v\" (UniqueName: \"kubernetes.io/projected/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-kube-api-access-lpm9v\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:59.841275 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.841025    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:59.841275 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.841041    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:59.841275 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.841055    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:59.841275 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.841068    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:59.841275 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.841114    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:48:59.874423 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.874397    2560 scope.go:117] "RemoveContainer" containerID="d2f3a69ac002f709642d6b5acc51e6ce32685ff2709d7ffe6ea22241699171fe"
Jul 29 18:48:59.874803 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:48:59.874770    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f3a69ac002f709642d6b5acc51e6ce32685ff2709d7ffe6ea22241699171fe\": container with ID starting with d2f3a69ac002f709642d6b5acc51e6ce32685ff2709d7ffe6ea22241699171fe not found: ID does not exist" containerID="d2f3a69ac002f709642d6b5acc51e6ce32685ff2709d7ffe6ea22241699171fe"
Jul 29 18:48:59.874905 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.874817    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f3a69ac002f709642d6b5acc51e6ce32685ff2709d7ffe6ea22241699171fe"} err="failed to get container status \"d2f3a69ac002f709642d6b5acc51e6ce32685ff2709d7ffe6ea22241699171fe\": rpc error: code = NotFound desc = could not find container \"d2f3a69ac002f709642d6b5acc51e6ce32685ff2709d7ffe6ea22241699171fe\": container with ID starting with d2f3a69ac002f709642d6b5acc51e6ce32685ff2709d7ffe6ea22241699171fe not found: ID does not exist"
Jul 29 18:48:59.874905 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.874846    2560 scope.go:117] "RemoveContainer" containerID="a30f113662381e1b266a68d841968bd108376100eba59ddf2f45504a17b222dc"
Jul 29 18:48:59.875129 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:48:59.875103    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30f113662381e1b266a68d841968bd108376100eba59ddf2f45504a17b222dc\": container with ID starting with a30f113662381e1b266a68d841968bd108376100eba59ddf2f45504a17b222dc not found: ID does not exist" containerID="a30f113662381e1b266a68d841968bd108376100eba59ddf2f45504a17b222dc"
Jul 29 18:48:59.875178 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:48:59.875138    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30f113662381e1b266a68d841968bd108376100eba59ddf2f45504a17b222dc"} err="failed to get container status \"a30f113662381e1b266a68d841968bd108376100eba59ddf2f45504a17b222dc\": rpc error: code = NotFound desc = could not find container \"a30f113662381e1b266a68d841968bd108376100eba59ddf2f45504a17b222dc\": container with ID starting with a30f113662381e1b266a68d841968bd108376100eba59ddf2f45504a17b222dc not found: ID does not exist"
Jul 29 18:49:00.107062 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.107026    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"]
Jul 29 18:49:00.111389 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.111363    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-c7457f4fd-7sjh6"]
Jul 29 18:49:00.553883 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.553834    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:49:00.649230 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.649118    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5c5069-8943-424d-a642-73377008824e-tls-certs\") pod \"de5c5069-8943-424d-a642-73377008824e\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") "
Jul 29 18:49:00.649230 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.649186    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-uds\") pod \"de5c5069-8943-424d-a642-73377008824e\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") "
Jul 29 18:49:00.649230 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.649220    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-tmp\") pod \"de5c5069-8943-424d-a642-73377008824e\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") "
Jul 29 18:49:00.649593 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.649240    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq8jv\" (UniqueName: \"kubernetes.io/projected/de5c5069-8943-424d-a642-73377008824e-kube-api-access-jq8jv\") pod \"de5c5069-8943-424d-a642-73377008824e\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") "
Jul 29 18:49:00.649593 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.649273    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-kserve-provision-location\") pod \"de5c5069-8943-424d-a642-73377008824e\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") "
Jul 29 18:49:00.649593 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.649298    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-cache\") pod \"de5c5069-8943-424d-a642-73377008824e\" (UID: \"de5c5069-8943-424d-a642-73377008824e\") "
Jul 29 18:49:00.649593 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.649483    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "de5c5069-8943-424d-a642-73377008824e" (UID: "de5c5069-8943-424d-a642-73377008824e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:49:00.649735 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.649591    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "de5c5069-8943-424d-a642-73377008824e" (UID: "de5c5069-8943-424d-a642-73377008824e"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:49:00.649735 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.649626    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "de5c5069-8943-424d-a642-73377008824e" (UID: "de5c5069-8943-424d-a642-73377008824e"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:49:00.649992 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.649971    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de5c5069-8943-424d-a642-73377008824e" (UID: "de5c5069-8943-424d-a642-73377008824e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:49:00.651315 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.651295    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de5c5069-8943-424d-a642-73377008824e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de5c5069-8943-424d-a642-73377008824e" (UID: "de5c5069-8943-424d-a642-73377008824e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:49:00.651420 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.651352    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5c5069-8943-424d-a642-73377008824e-kube-api-access-jq8jv" (OuterVolumeSpecName: "kube-api-access-jq8jv") pod "de5c5069-8943-424d-a642-73377008824e" (UID: "de5c5069-8943-424d-a642-73377008824e"). InnerVolumeSpecName "kube-api-access-jq8jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:49:00.687361 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.687333    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" path="/var/lib/kubelet/pods/733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4/volumes"
Jul 29 18:49:00.750450 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.750420    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5c5069-8943-424d-a642-73377008824e-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:00.750450 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.750446    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-uds\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:00.750450 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.750455    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-tmp\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:00.750645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.750465    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq8jv\" (UniqueName: \"kubernetes.io/projected/de5c5069-8943-424d-a642-73377008824e-kube-api-access-jq8jv\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:00.750645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.750479    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:00.750645 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.750489    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5c5069-8943-424d-a642-73377008824e-tokenizer-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:00.790883 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.790857    2560 generic.go:356] "Generic (PLEG): container finished" podID="de5c5069-8943-424d-a642-73377008824e" containerID="7d7e93c669c0e313d0a587d2d2c2775ffe4535e1a67920af466ec9ef165542e2" exitCode=0
Jul 29 18:49:00.791029 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.790931    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"
Jul 29 18:49:00.791029 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.790937    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m" event={"ID":"de5c5069-8943-424d-a642-73377008824e","Type":"ContainerDied","Data":"7d7e93c669c0e313d0a587d2d2c2775ffe4535e1a67920af466ec9ef165542e2"}
Jul 29 18:49:00.791029 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.791003    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m" event={"ID":"de5c5069-8943-424d-a642-73377008824e","Type":"ContainerDied","Data":"df0196fa2342744c2892266a6f3c28fe40e151c43acff0bc7af79c9f8ca6855d"}
Jul 29 18:49:00.791161 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.791037    2560 scope.go:117] "RemoveContainer" containerID="7d7e93c669c0e313d0a587d2d2c2775ffe4535e1a67920af466ec9ef165542e2"
Jul 29 18:49:00.800178 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.800149    2560 scope.go:117] "RemoveContainer" containerID="54a98a1d209d7b7fa977b99a9fbd9b9088da509a45c4b474505fbf0a0de2cb7f"
Jul 29 18:49:00.808041 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.808023    2560 scope.go:117] "RemoveContainer" containerID="5f82308ae468c87caa72fee5e923f2cbea16c8528b28b3abff51185e59eb8f8c"
Jul 29 18:49:00.809801 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.809783    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"]
Jul 29 18:49:00.815922 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.815902    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-856ddff9nw6m"]
Jul 29 18:49:00.816137 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.816124    2560 scope.go:117] "RemoveContainer" containerID="7d7e93c669c0e313d0a587d2d2c2775ffe4535e1a67920af466ec9ef165542e2"
Jul 29 18:49:00.816394 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:49:00.816378    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7e93c669c0e313d0a587d2d2c2775ffe4535e1a67920af466ec9ef165542e2\": container with ID starting with 7d7e93c669c0e313d0a587d2d2c2775ffe4535e1a67920af466ec9ef165542e2 not found: ID does not exist" containerID="7d7e93c669c0e313d0a587d2d2c2775ffe4535e1a67920af466ec9ef165542e2"
Jul 29 18:49:00.816453 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.816400    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7e93c669c0e313d0a587d2d2c2775ffe4535e1a67920af466ec9ef165542e2"} err="failed to get container status \"7d7e93c669c0e313d0a587d2d2c2775ffe4535e1a67920af466ec9ef165542e2\": rpc error: code = NotFound desc = could not find container \"7d7e93c669c0e313d0a587d2d2c2775ffe4535e1a67920af466ec9ef165542e2\": container with ID starting with 7d7e93c669c0e313d0a587d2d2c2775ffe4535e1a67920af466ec9ef165542e2 not found: ID does not exist"
Jul 29 18:49:00.816453 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.816417    2560 scope.go:117] "RemoveContainer" containerID="54a98a1d209d7b7fa977b99a9fbd9b9088da509a45c4b474505fbf0a0de2cb7f"
Jul 29 18:49:00.816667 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:49:00.816650    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a98a1d209d7b7fa977b99a9fbd9b9088da509a45c4b474505fbf0a0de2cb7f\": container with ID starting with 54a98a1d209d7b7fa977b99a9fbd9b9088da509a45c4b474505fbf0a0de2cb7f not found: ID does not exist" containerID="54a98a1d209d7b7fa977b99a9fbd9b9088da509a45c4b474505fbf0a0de2cb7f"
Jul 29 18:49:00.816708 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.816671    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a98a1d209d7b7fa977b99a9fbd9b9088da509a45c4b474505fbf0a0de2cb7f"} err="failed to get container status \"54a98a1d209d7b7fa977b99a9fbd9b9088da509a45c4b474505fbf0a0de2cb7f\": rpc error: code = NotFound desc = could not find container \"54a98a1d209d7b7fa977b99a9fbd9b9088da509a45c4b474505fbf0a0de2cb7f\": container with ID starting with 54a98a1d209d7b7fa977b99a9fbd9b9088da509a45c4b474505fbf0a0de2cb7f not found: ID does not exist"
Jul 29 18:49:00.816708 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.816684    2560 scope.go:117] "RemoveContainer" containerID="5f82308ae468c87caa72fee5e923f2cbea16c8528b28b3abff51185e59eb8f8c"
Jul 29 18:49:00.816900 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:49:00.816885    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f82308ae468c87caa72fee5e923f2cbea16c8528b28b3abff51185e59eb8f8c\": container with ID starting with 5f82308ae468c87caa72fee5e923f2cbea16c8528b28b3abff51185e59eb8f8c not found: ID does not exist" containerID="5f82308ae468c87caa72fee5e923f2cbea16c8528b28b3abff51185e59eb8f8c"
Jul 29 18:49:00.816941 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:00.816905    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f82308ae468c87caa72fee5e923f2cbea16c8528b28b3abff51185e59eb8f8c"} err="failed to get container status \"5f82308ae468c87caa72fee5e923f2cbea16c8528b28b3abff51185e59eb8f8c\": rpc error: code = NotFound desc = could not find container \"5f82308ae468c87caa72fee5e923f2cbea16c8528b28b3abff51185e59eb8f8c\": container with ID starting with 5f82308ae468c87caa72fee5e923f2cbea16c8528b28b3abff51185e59eb8f8c not found: ID does not exist"
Jul 29 18:49:02.688057 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:02.688022    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de5c5069-8943-424d-a642-73377008824e" path="/var/lib/kubelet/pods/de5c5069-8943-424d-a642-73377008824e/volumes"
Jul 29 18:49:08.441598 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.441541    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"]
Jul 29 18:49:08.442079 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.442025    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5c5069-8943-424d-a642-73377008824e" containerName="tokenizer"
Jul 29 18:49:08.442079 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.442045    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5c5069-8943-424d-a642-73377008824e" containerName="tokenizer"
Jul 29 18:49:08.442079 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.442055    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5c5069-8943-424d-a642-73377008824e" containerName="main"
Jul 29 18:49:08.442079 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.442061    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5c5069-8943-424d-a642-73377008824e" containerName="main"
Jul 29 18:49:08.442079 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.442076    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5c5069-8943-424d-a642-73377008824e" containerName="storage-initializer"
Jul 29 18:49:08.442079 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.442082    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5c5069-8943-424d-a642-73377008824e" containerName="storage-initializer"
Jul 29 18:49:08.442318 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.442089    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" containerName="main"
Jul 29 18:49:08.442318 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.442094    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" containerName="main"
Jul 29 18:49:08.442318 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.442109    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" containerName="storage-initializer"
Jul 29 18:49:08.442318 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.442117    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" containerName="storage-initializer"
Jul 29 18:49:08.442318 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.442168    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="733a9f9b-8e2d-45e7-8ecb-3d3c1b7c2aa4" containerName="main"
Jul 29 18:49:08.442318 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.442176    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="de5c5069-8943-424d-a642-73377008824e" containerName="main"
Jul 29 18:49:08.442318 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.442183    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="de5c5069-8943-424d-a642-73377008824e" containerName="tokenizer"
Jul 29 18:49:08.445604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.445583    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.448356 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.448333    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 29 18:49:08.449435 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.449419    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2cmt9\""
Jul 29 18:49:08.456323 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.456302    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"]
Jul 29 18:49:08.480012 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.479980    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:49:08.620315 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.620281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.620488 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.620322    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-dshm\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.620488 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.620356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-tls-certs\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.620488 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.620379    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-home\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.620488 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.620399    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgw9f\" (UniqueName: \"kubernetes.io/projected/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-kube-api-access-vgw9f\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.620488 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.620445    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-tmp-dir\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.620488 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.620464    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-model-cache\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.707589 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.707511    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"]
Jul 29 18:49:08.710386 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.710362    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.713038 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.713016    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-fvjsp\""
Jul 29 18:49:08.721774 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.721754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.721895 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.721785    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-dshm\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.721895 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.721805    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-tls-certs\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.721895 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.721822    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-home\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.721895 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.721837    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgw9f\" (UniqueName: \"kubernetes.io/projected/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-kube-api-access-vgw9f\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.721895 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.721861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-tmp-dir\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.721895 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.721881    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-model-cache\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.722312 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.722128    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.722312 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.722158    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-home\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.722312 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.722285    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-tmp-dir\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.722312 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.722295    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-model-cache\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.724016 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.723996    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-dshm\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.724455 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.724438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-tls-certs\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.728895 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.728870    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"]
Jul 29 18:49:08.733169 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.733145    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgw9f\" (UniqueName: \"kubernetes.io/projected/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-kube-api-access-vgw9f\") pod \"precise-prefix-cache-test-kserve-79764f4d79-28xrv\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.738903 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.738873    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:49:08.756269 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.756246    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:08.822887 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.822496    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.822887 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.822573    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.822887 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.822605    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxjjk\" (UniqueName: \"kubernetes.io/projected/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-kube-api-access-lxjjk\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.822887 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.822634    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.822887 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.822668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.822887 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.822723    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.923655 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.922676    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"]
Jul 29 18:49:08.931635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.927170    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.931635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.927514    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.931635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.927596    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.931635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.927626    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxjjk\" (UniqueName: \"kubernetes.io/projected/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-kube-api-access-lxjjk\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.931635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.927662    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.931635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.927699    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.931635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.928763    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.931635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.929023    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.931635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.929060    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.931635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.929558    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.932472 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:49:08.932448    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6d57cf_546b_4d5f_9dd8_64ea4549e477.slice/crio-417cebcf5631005ed039caec339a14a6f08325e2f8220e479d59f8e974325d77 WatchSource:0}: Error finding container 417cebcf5631005ed039caec339a14a6f08325e2f8220e479d59f8e974325d77: Status 404 returned error can't find the container with id 417cebcf5631005ed039caec339a14a6f08325e2f8220e479d59f8e974325d77
Jul 29 18:49:08.933793 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.933771    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:08.938371 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:08.938357    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxjjk\" (UniqueName: \"kubernetes.io/projected/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-kube-api-access-lxjjk\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:09.020747 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:09.020715    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:09.165609 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:09.165574    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"]
Jul 29 18:49:09.169543 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:49:09.169498    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cc7cc6_a527_4a5e_9d8d_1a07965c7170.slice/crio-5d82eb65fd35f1973f27c96cd8fd88361a7403f40de20fe21901c5fcb829d6b3 WatchSource:0}: Error finding container 5d82eb65fd35f1973f27c96cd8fd88361a7403f40de20fe21901c5fcb829d6b3: Status 404 returned error can't find the container with id 5d82eb65fd35f1973f27c96cd8fd88361a7403f40de20fe21901c5fcb829d6b3
Jul 29 18:49:09.845541 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:09.845504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w" event={"ID":"91cc7cc6-a527-4a5e-9d8d-1a07965c7170","Type":"ContainerStarted","Data":"c4b640f0fba7fe4bf3fc9bd458f49d27b39eccf2aa784050c77d6fd763bae5d0"}
Jul 29 18:49:09.845991 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:09.845568    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w" event={"ID":"91cc7cc6-a527-4a5e-9d8d-1a07965c7170","Type":"ContainerStarted","Data":"5d82eb65fd35f1973f27c96cd8fd88361a7403f40de20fe21901c5fcb829d6b3"}
Jul 29 18:49:09.847115 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:09.847079    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv" event={"ID":"0b6d57cf-546b-4d5f-9dd8-64ea4549e477","Type":"ContainerStarted","Data":"3d77ae41c665b176d7a6e87d318b845bb311222ce0f90b7ba14527590b952a39"}
Jul 29 18:49:09.847270 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:09.847142    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv" event={"ID":"0b6d57cf-546b-4d5f-9dd8-64ea4549e477","Type":"ContainerStarted","Data":"417cebcf5631005ed039caec339a14a6f08325e2f8220e479d59f8e974325d77"}
Jul 29 18:49:10.855548 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:10.855470    2560 generic.go:356] "Generic (PLEG): container finished" podID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerID="c4b640f0fba7fe4bf3fc9bd458f49d27b39eccf2aa784050c77d6fd763bae5d0" exitCode=0
Jul 29 18:49:10.855548 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:10.855560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w" event={"ID":"91cc7cc6-a527-4a5e-9d8d-1a07965c7170","Type":"ContainerDied","Data":"c4b640f0fba7fe4bf3fc9bd458f49d27b39eccf2aa784050c77d6fd763bae5d0"}
Jul 29 18:49:11.865123 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:11.865057    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w" event={"ID":"91cc7cc6-a527-4a5e-9d8d-1a07965c7170","Type":"ContainerStarted","Data":"4687990fa1c96d1d369aefb736dcc2ab1d9ee84920d00927cb8a2509c06a8d4d"}
Jul 29 18:49:11.865601 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:11.865134    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w" event={"ID":"91cc7cc6-a527-4a5e-9d8d-1a07965c7170","Type":"ContainerStarted","Data":"07e02c670e067e6f9defc3d609cdc67fc37f37f119b691e23c0b2957a2761ded"}
Jul 29 18:49:11.865601 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:11.865185    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:11.891870 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:11.891800    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w" podStartSLOduration=3.090424286 podStartE2EDuration="3.891774034s" totalImagesPullingTime="0s" totalInitContainerRuntime="801.349748ms" isStatefulPod=false podCreationTimestamp="2026-07-29 18:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:49:11.886589329 +0000 UTC m=+877.780589750" watchObservedRunningTime="2026-07-29 18:49:11.891774034 +0000 UTC m=+877.785774436"
Jul 29 18:49:12.407659 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:12.407626    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:49:15.891064 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:15.891030    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b6d57cf-546b-4d5f-9dd8-64ea4549e477" containerID="3d77ae41c665b176d7a6e87d318b845bb311222ce0f90b7ba14527590b952a39" exitCode=0
Jul 29 18:49:15.891478 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:15.891101    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv" event={"ID":"0b6d57cf-546b-4d5f-9dd8-64ea4549e477","Type":"ContainerDied","Data":"3d77ae41c665b176d7a6e87d318b845bb311222ce0f90b7ba14527590b952a39"}
Jul 29 18:49:16.897452 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:16.897423    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv" event={"ID":"0b6d57cf-546b-4d5f-9dd8-64ea4549e477","Type":"ContainerStarted","Data":"444a43b3df1511a2251996d70cae2dcd12f19c3388d922fb0ecd8bc4ad46aca8"}
Jul 29 18:49:16.939842 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:16.939794    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv" podStartSLOduration=2.478597922 podStartE2EDuration="8.939778601s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.461180679s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:49:16.937309618 +0000 UTC m=+882.831310040" watchObservedRunningTime="2026-07-29 18:49:16.939778601 +0000 UTC m=+882.833779001"
Jul 29 18:49:18.757376 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:18.757334    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:18.757839 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:18.757414    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:18.770408 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:18.770378    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:18.918953 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:18.918918    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:19.021901 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:19.021819    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:19.021901 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:19.021885    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:19.023056 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:49:19.023033    2560 logging.go:55] [core] [Channel #62 SubChannel #63]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.46:9003", ServerName: "10.133.0.46:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.46:9003: connect: connection refused"
Jul 29 18:49:19.024320 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:19.024301    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:19.914158 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:19.914127    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:20.022441 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:20.022386    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w" podUID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.46:9003\" within 1s: context deadline exceeded"
Jul 29 18:49:29.022383 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:49:29.022302    2560 logging.go:55] [core] [Channel #64 SubChannel #65]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.46:9003", ServerName: "10.133.0.46:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.46:9003: connect: connection refused"
Jul 29 18:49:30.022612 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:30.022562    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w" podUID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.46:9003\" within 1s: context deadline exceeded"
Jul 29 18:49:30.022998 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:49:30.022680    2560 logging.go:55] [core] [Channel #64 SubChannel #65]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.46:9003", ServerName: "10.133.0.46:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.46:9003: operation was canceled"
Jul 29 18:49:34.527907 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:34.527874    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 18:49:34.528421 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:34.528386    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 18:49:40.919828 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:40.919798    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:41.972235 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:41.972185    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"]
Jul 29 18:49:41.972710 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:41.972466    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv" podUID="0b6d57cf-546b-4d5f-9dd8-64ea4549e477" containerName="main" containerID="cri-o://444a43b3df1511a2251996d70cae2dcd12f19c3388d922fb0ecd8bc4ad46aca8" gracePeriod=30
Jul 29 18:49:41.986296 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:41.986261    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"]
Jul 29 18:49:41.986647 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:41.986615    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w" podUID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerName="main" containerID="cri-o://07e02c670e067e6f9defc3d609cdc67fc37f37f119b691e23c0b2957a2761ded" gracePeriod=30
Jul 29 18:49:41.986917 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:41.986889    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w" podUID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerName="tokenizer" containerID="cri-o://4687990fa1c96d1d369aefb736dcc2ab1d9ee84920d00927cb8a2509c06a8d4d" gracePeriod=30
Jul 29 18:49:42.022906 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.022871    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:49:42.142900 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.142866    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:49:42.232508 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.232441    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:42.318865 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.318833    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgw9f\" (UniqueName: \"kubernetes.io/projected/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-kube-api-access-vgw9f\") pod \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") "
Jul 29 18:49:42.319032 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.318925    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-tmp-dir\") pod \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") "
Jul 29 18:49:42.319032 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.318968    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-model-cache\") pod \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") "
Jul 29 18:49:42.319032 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.319009    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-tls-certs\") pod \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") "
Jul 29 18:49:42.319176 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.319064    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-dshm\") pod \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") "
Jul 29 18:49:42.319176 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.319094    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-home\") pod \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") "
Jul 29 18:49:42.319176 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.319126    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-kserve-provision-location\") pod \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\" (UID: \"0b6d57cf-546b-4d5f-9dd8-64ea4549e477\") "
Jul 29 18:49:42.319372 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.319258    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b6d57cf-546b-4d5f-9dd8-64ea4549e477" (UID: "0b6d57cf-546b-4d5f-9dd8-64ea4549e477"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:49:42.319372 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.319266    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b6d57cf-546b-4d5f-9dd8-64ea4549e477" (UID: "0b6d57cf-546b-4d5f-9dd8-64ea4549e477"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:49:42.319490 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.319454    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-home" (OuterVolumeSpecName: "home") pod "0b6d57cf-546b-4d5f-9dd8-64ea4549e477" (UID: "0b6d57cf-546b-4d5f-9dd8-64ea4549e477"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:49:42.319490 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.319473    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:42.319593 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.319495    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:42.321209 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.321172    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-kube-api-access-vgw9f" (OuterVolumeSpecName: "kube-api-access-vgw9f") pod "0b6d57cf-546b-4d5f-9dd8-64ea4549e477" (UID: "0b6d57cf-546b-4d5f-9dd8-64ea4549e477"). InnerVolumeSpecName "kube-api-access-vgw9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:49:42.321480 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.321444    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-dshm" (OuterVolumeSpecName: "dshm") pod "0b6d57cf-546b-4d5f-9dd8-64ea4549e477" (UID: "0b6d57cf-546b-4d5f-9dd8-64ea4549e477"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:49:42.321614 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.321590    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b6d57cf-546b-4d5f-9dd8-64ea4549e477" (UID: "0b6d57cf-546b-4d5f-9dd8-64ea4549e477"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:49:42.376703 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.376633    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b6d57cf-546b-4d5f-9dd8-64ea4549e477" (UID: "0b6d57cf-546b-4d5f-9dd8-64ea4549e477"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:49:42.420383 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.420339    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:42.420383 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.420370    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:42.420383 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.420381    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:42.420383 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.420392    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgw9f\" (UniqueName: \"kubernetes.io/projected/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-kube-api-access-vgw9f\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:42.420707 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:42.420402    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6d57cf-546b-4d5f-9dd8-64ea4549e477-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:43.043347 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.043254    2560 generic.go:356] "Generic (PLEG): container finished" podID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerID="07e02c670e067e6f9defc3d609cdc67fc37f37f119b691e23c0b2957a2761ded" exitCode=0
Jul 29 18:49:43.043347 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.043324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w" event={"ID":"91cc7cc6-a527-4a5e-9d8d-1a07965c7170","Type":"ContainerDied","Data":"07e02c670e067e6f9defc3d609cdc67fc37f37f119b691e23c0b2957a2761ded"}
Jul 29 18:49:43.044938 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.044911    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b6d57cf-546b-4d5f-9dd8-64ea4549e477" containerID="444a43b3df1511a2251996d70cae2dcd12f19c3388d922fb0ecd8bc4ad46aca8" exitCode=0
Jul 29 18:49:43.045094 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.044968    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"
Jul 29 18:49:43.045094 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.044998    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv" event={"ID":"0b6d57cf-546b-4d5f-9dd8-64ea4549e477","Type":"ContainerDied","Data":"444a43b3df1511a2251996d70cae2dcd12f19c3388d922fb0ecd8bc4ad46aca8"}
Jul 29 18:49:43.045094 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.045045    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv" event={"ID":"0b6d57cf-546b-4d5f-9dd8-64ea4549e477","Type":"ContainerDied","Data":"417cebcf5631005ed039caec339a14a6f08325e2f8220e479d59f8e974325d77"}
Jul 29 18:49:43.045094 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.045069    2560 scope.go:117] "RemoveContainer" containerID="444a43b3df1511a2251996d70cae2dcd12f19c3388d922fb0ecd8bc4ad46aca8"
Jul 29 18:49:43.054340 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.054320    2560 scope.go:117] "RemoveContainer" containerID="3d77ae41c665b176d7a6e87d318b845bb311222ce0f90b7ba14527590b952a39"
Jul 29 18:49:43.063553 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.063528    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"]
Jul 29 18:49:43.069627 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.069598    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-28xrv"]
Jul 29 18:49:43.122137 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.122111    2560 scope.go:117] "RemoveContainer" containerID="444a43b3df1511a2251996d70cae2dcd12f19c3388d922fb0ecd8bc4ad46aca8"
Jul 29 18:49:43.122597 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:49:43.122570    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"444a43b3df1511a2251996d70cae2dcd12f19c3388d922fb0ecd8bc4ad46aca8\": container with ID starting with 444a43b3df1511a2251996d70cae2dcd12f19c3388d922fb0ecd8bc4ad46aca8 not found: ID does not exist" containerID="444a43b3df1511a2251996d70cae2dcd12f19c3388d922fb0ecd8bc4ad46aca8"
Jul 29 18:49:43.122663 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.122612    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444a43b3df1511a2251996d70cae2dcd12f19c3388d922fb0ecd8bc4ad46aca8"} err="failed to get container status \"444a43b3df1511a2251996d70cae2dcd12f19c3388d922fb0ecd8bc4ad46aca8\": rpc error: code = NotFound desc = could not find container \"444a43b3df1511a2251996d70cae2dcd12f19c3388d922fb0ecd8bc4ad46aca8\": container with ID starting with 444a43b3df1511a2251996d70cae2dcd12f19c3388d922fb0ecd8bc4ad46aca8 not found: ID does not exist"
Jul 29 18:49:43.122663 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.122639    2560 scope.go:117] "RemoveContainer" containerID="3d77ae41c665b176d7a6e87d318b845bb311222ce0f90b7ba14527590b952a39"
Jul 29 18:49:43.122969 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:49:43.122938    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d77ae41c665b176d7a6e87d318b845bb311222ce0f90b7ba14527590b952a39\": container with ID starting with 3d77ae41c665b176d7a6e87d318b845bb311222ce0f90b7ba14527590b952a39 not found: ID does not exist" containerID="3d77ae41c665b176d7a6e87d318b845bb311222ce0f90b7ba14527590b952a39"
Jul 29 18:49:43.123104 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.122972    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d77ae41c665b176d7a6e87d318b845bb311222ce0f90b7ba14527590b952a39"} err="failed to get container status \"3d77ae41c665b176d7a6e87d318b845bb311222ce0f90b7ba14527590b952a39\": rpc error: code = NotFound desc = could not find container \"3d77ae41c665b176d7a6e87d318b845bb311222ce0f90b7ba14527590b952a39\": container with ID starting with 3d77ae41c665b176d7a6e87d318b845bb311222ce0f90b7ba14527590b952a39 not found: ID does not exist"
Jul 29 18:49:43.555066 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.555043    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:43.633874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.633843    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-uds\") pod \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") "
Jul 29 18:49:43.634044 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.633913    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tls-certs\") pod \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") "
Jul 29 18:49:43.634044 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.633950    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-kserve-provision-location\") pod \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") "
Jul 29 18:49:43.634118 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.634047    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxjjk\" (UniqueName: \"kubernetes.io/projected/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-kube-api-access-lxjjk\") pod \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") "
Jul 29 18:49:43.634118 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.634088    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-cache\") pod \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") "
Jul 29 18:49:43.634181 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.634126    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-tmp\") pod \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\" (UID: \"91cc7cc6-a527-4a5e-9d8d-1a07965c7170\") "
Jul 29 18:49:43.634257 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.634176    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "91cc7cc6-a527-4a5e-9d8d-1a07965c7170" (UID: "91cc7cc6-a527-4a5e-9d8d-1a07965c7170"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:49:43.634469 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.634443    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-uds\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:43.634598 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.634510    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "91cc7cc6-a527-4a5e-9d8d-1a07965c7170" (UID: "91cc7cc6-a527-4a5e-9d8d-1a07965c7170"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:49:43.634598 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.634504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "91cc7cc6-a527-4a5e-9d8d-1a07965c7170" (UID: "91cc7cc6-a527-4a5e-9d8d-1a07965c7170"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:49:43.634719 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.634693    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "91cc7cc6-a527-4a5e-9d8d-1a07965c7170" (UID: "91cc7cc6-a527-4a5e-9d8d-1a07965c7170"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:49:43.636146 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.636119    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91cc7cc6-a527-4a5e-9d8d-1a07965c7170" (UID: "91cc7cc6-a527-4a5e-9d8d-1a07965c7170"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:49:43.636146 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.636134    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-kube-api-access-lxjjk" (OuterVolumeSpecName: "kube-api-access-lxjjk") pod "91cc7cc6-a527-4a5e-9d8d-1a07965c7170" (UID: "91cc7cc6-a527-4a5e-9d8d-1a07965c7170"). InnerVolumeSpecName "kube-api-access-lxjjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:49:43.735597 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.735558    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:43.735597 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.735591    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:43.735597 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.735603    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxjjk\" (UniqueName: \"kubernetes.io/projected/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-kube-api-access-lxjjk\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:43.735829 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.735611    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:43.735829 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:43.735621    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/91cc7cc6-a527-4a5e-9d8d-1a07965c7170-tokenizer-tmp\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:49:44.051996 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.051913    2560 generic.go:356] "Generic (PLEG): container finished" podID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerID="4687990fa1c96d1d369aefb736dcc2ab1d9ee84920d00927cb8a2509c06a8d4d" exitCode=0
Jul 29 18:49:44.051996 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.051982    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"
Jul 29 18:49:44.052512 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.052003    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w" event={"ID":"91cc7cc6-a527-4a5e-9d8d-1a07965c7170","Type":"ContainerDied","Data":"4687990fa1c96d1d369aefb736dcc2ab1d9ee84920d00927cb8a2509c06a8d4d"}
Jul 29 18:49:44.052512 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.052068    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w" event={"ID":"91cc7cc6-a527-4a5e-9d8d-1a07965c7170","Type":"ContainerDied","Data":"5d82eb65fd35f1973f27c96cd8fd88361a7403f40de20fe21901c5fcb829d6b3"}
Jul 29 18:49:44.052512 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.052100    2560 scope.go:117] "RemoveContainer" containerID="4687990fa1c96d1d369aefb736dcc2ab1d9ee84920d00927cb8a2509c06a8d4d"
Jul 29 18:49:44.061462 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.061442    2560 scope.go:117] "RemoveContainer" containerID="07e02c670e067e6f9defc3d609cdc67fc37f37f119b691e23c0b2957a2761ded"
Jul 29 18:49:44.069569 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.069551    2560 scope.go:117] "RemoveContainer" containerID="c4b640f0fba7fe4bf3fc9bd458f49d27b39eccf2aa784050c77d6fd763bae5d0"
Jul 29 18:49:44.075003 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.074979    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"]
Jul 29 18:49:44.077680 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.077657    2560 scope.go:117] "RemoveContainer" containerID="4687990fa1c96d1d369aefb736dcc2ab1d9ee84920d00927cb8a2509c06a8d4d"
Jul 29 18:49:44.077927 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:49:44.077907    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4687990fa1c96d1d369aefb736dcc2ab1d9ee84920d00927cb8a2509c06a8d4d\": container with ID starting with 4687990fa1c96d1d369aefb736dcc2ab1d9ee84920d00927cb8a2509c06a8d4d not found: ID does not exist" containerID="4687990fa1c96d1d369aefb736dcc2ab1d9ee84920d00927cb8a2509c06a8d4d"
Jul 29 18:49:44.077978 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.077937    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4687990fa1c96d1d369aefb736dcc2ab1d9ee84920d00927cb8a2509c06a8d4d"} err="failed to get container status \"4687990fa1c96d1d369aefb736dcc2ab1d9ee84920d00927cb8a2509c06a8d4d\": rpc error: code = NotFound desc = could not find container \"4687990fa1c96d1d369aefb736dcc2ab1d9ee84920d00927cb8a2509c06a8d4d\": container with ID starting with 4687990fa1c96d1d369aefb736dcc2ab1d9ee84920d00927cb8a2509c06a8d4d not found: ID does not exist"
Jul 29 18:49:44.077978 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.077954    2560 scope.go:117] "RemoveContainer" containerID="07e02c670e067e6f9defc3d609cdc67fc37f37f119b691e23c0b2957a2761ded"
Jul 29 18:49:44.078164 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:49:44.078149    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e02c670e067e6f9defc3d609cdc67fc37f37f119b691e23c0b2957a2761ded\": container with ID starting with 07e02c670e067e6f9defc3d609cdc67fc37f37f119b691e23c0b2957a2761ded not found: ID does not exist" containerID="07e02c670e067e6f9defc3d609cdc67fc37f37f119b691e23c0b2957a2761ded"
Jul 29 18:49:44.078235 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.078168    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e02c670e067e6f9defc3d609cdc67fc37f37f119b691e23c0b2957a2761ded"} err="failed to get container status \"07e02c670e067e6f9defc3d609cdc67fc37f37f119b691e23c0b2957a2761ded\": rpc error: code = NotFound desc = could not find container \"07e02c670e067e6f9defc3d609cdc67fc37f37f119b691e23c0b2957a2761ded\": container with ID starting with 07e02c670e067e6f9defc3d609cdc67fc37f37f119b691e23c0b2957a2761ded not found: ID does not exist"
Jul 29 18:49:44.078235 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.078181    2560 scope.go:117] "RemoveContainer" containerID="c4b640f0fba7fe4bf3fc9bd458f49d27b39eccf2aa784050c77d6fd763bae5d0"
Jul 29 18:49:44.078437 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:49:44.078400    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b640f0fba7fe4bf3fc9bd458f49d27b39eccf2aa784050c77d6fd763bae5d0\": container with ID starting with c4b640f0fba7fe4bf3fc9bd458f49d27b39eccf2aa784050c77d6fd763bae5d0 not found: ID does not exist" containerID="c4b640f0fba7fe4bf3fc9bd458f49d27b39eccf2aa784050c77d6fd763bae5d0"
Jul 29 18:49:44.078478 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.078445    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b640f0fba7fe4bf3fc9bd458f49d27b39eccf2aa784050c77d6fd763bae5d0"} err="failed to get container status \"c4b640f0fba7fe4bf3fc9bd458f49d27b39eccf2aa784050c77d6fd763bae5d0\": rpc error: code = NotFound desc = could not find container \"c4b640f0fba7fe4bf3fc9bd458f49d27b39eccf2aa784050c77d6fd763bae5d0\": container with ID starting with c4b640f0fba7fe4bf3fc9bd458f49d27b39eccf2aa784050c77d6fd763bae5d0 not found: ID does not exist"
Jul 29 18:49:44.083518 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.083497    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5c784f78l2w7w"]
Jul 29 18:49:44.688452 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.688410    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b6d57cf-546b-4d5f-9dd8-64ea4549e477" path="/var/lib/kubelet/pods/0b6d57cf-546b-4d5f-9dd8-64ea4549e477/volumes"
Jul 29 18:49:44.688935 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:44.688914    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" path="/var/lib/kubelet/pods/91cc7cc6-a527-4a5e-9d8d-1a07965c7170/volumes"
Jul 29 18:49:50.667793 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.667761    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"]
Jul 29 18:49:50.668273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.668099    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerName="storage-initializer"
Jul 29 18:49:50.668273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.668112    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerName="storage-initializer"
Jul 29 18:49:50.668273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.668119    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerName="tokenizer"
Jul 29 18:49:50.668273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.668125    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerName="tokenizer"
Jul 29 18:49:50.668273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.668139    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerName="main"
Jul 29 18:49:50.668273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.668144    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerName="main"
Jul 29 18:49:50.668273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.668155    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6d57cf-546b-4d5f-9dd8-64ea4549e477" containerName="storage-initializer"
Jul 29 18:49:50.668273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.668160    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6d57cf-546b-4d5f-9dd8-64ea4549e477" containerName="storage-initializer"
Jul 29 18:49:50.668273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.668170    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6d57cf-546b-4d5f-9dd8-64ea4549e477" containerName="main"
Jul 29 18:49:50.668273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.668175    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6d57cf-546b-4d5f-9dd8-64ea4549e477" containerName="main"
Jul 29 18:49:50.668273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.668267    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b6d57cf-546b-4d5f-9dd8-64ea4549e477" containerName="main"
Jul 29 18:49:50.668273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.668276    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerName="main"
Jul 29 18:49:50.668686 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.668283    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="91cc7cc6-a527-4a5e-9d8d-1a07965c7170" containerName="tokenizer"
Jul 29 18:49:50.671370 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.671351    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.673899 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.673874    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 29 18:49:50.674014 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.673919    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2cmt9\""
Jul 29 18:49:50.680921 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.680897    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"]
Jul 29 18:49:50.743980 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.743947    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:49:50.797095 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.797059    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.797095 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.797091    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-kserve-provision-location\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.797407 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.797112    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-dshm\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.797407 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.797143    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-home\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.797407 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.797171    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmn9b\" (UniqueName: \"kubernetes.io/projected/6a78c984-72be-4b36-9d93-7aaffb889e46-kube-api-access-fmn9b\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.797407 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.797289    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-tmp-dir\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.797407 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.797354    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-model-cache\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.898522 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.898479    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-model-cache\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.898706 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.898556    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.898706 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.898582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-kserve-provision-location\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.898706 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.898605    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-dshm\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.898870 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.898726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-home\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.898870 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.898769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmn9b\" (UniqueName: \"kubernetes.io/projected/6a78c984-72be-4b36-9d93-7aaffb889e46-kube-api-access-fmn9b\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.898870 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.898854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-tmp-dir\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.899024 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.898925    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-model-cache\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.899024 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.898949    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-kserve-provision-location\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.899188 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.899165    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-home\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.899294 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.899188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-tmp-dir\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.900966 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.900943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-dshm\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.901145 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.901066    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.912273 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.912243    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmn9b\" (UniqueName: \"kubernetes.io/projected/6a78c984-72be-4b36-9d93-7aaffb889e46-kube-api-access-fmn9b\") pod \"stop-feature-test-kserve-6df78cd5bb-dq962\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:50.951635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.951552    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:49:50.982512 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:50.982483    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:49:51.120989 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:51.120959    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"]
Jul 29 18:49:51.123267 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:49:51.123239    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a78c984_72be_4b36_9d93_7aaffb889e46.slice/crio-3ccdbbf0b9b13cc385ce922a746754e9a13f503e7a3e0f012be46963e3a5c4e1 WatchSource:0}: Error finding container 3ccdbbf0b9b13cc385ce922a746754e9a13f503e7a3e0f012be46963e3a5c4e1: Status 404 returned error can't find the container with id 3ccdbbf0b9b13cc385ce922a746754e9a13f503e7a3e0f012be46963e3a5c4e1
Jul 29 18:49:52.101870 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:52.101809    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" event={"ID":"6a78c984-72be-4b36-9d93-7aaffb889e46","Type":"ContainerStarted","Data":"2abcad60ac46d4298b1c05e9e423e6b247b71ffb8065b374603cfa90c5dac73e"}
Jul 29 18:49:52.102486 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:52.101890    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" event={"ID":"6a78c984-72be-4b36-9d93-7aaffb889e46","Type":"ContainerStarted","Data":"3ccdbbf0b9b13cc385ce922a746754e9a13f503e7a3e0f012be46963e3a5c4e1"}
Jul 29 18:49:58.138534 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:58.138497    2560 generic.go:356] "Generic (PLEG): container finished" podID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerID="2abcad60ac46d4298b1c05e9e423e6b247b71ffb8065b374603cfa90c5dac73e" exitCode=0
Jul 29 18:49:58.138966 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:49:58.138545    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" event={"ID":"6a78c984-72be-4b36-9d93-7aaffb889e46","Type":"ContainerDied","Data":"2abcad60ac46d4298b1c05e9e423e6b247b71ffb8065b374603cfa90c5dac73e"}
Jul 29 18:50:22.423851 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:50:22.423817    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:50:38.381472 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:50:38.381435    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" event={"ID":"6a78c984-72be-4b36-9d93-7aaffb889e46","Type":"ContainerStarted","Data":"e92aedd12afab8f55cc1c4d7f9f972eaa11164dacbe9db1f173cdd015dde17ba"}
Jul 29 18:50:38.405319 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:50:38.405263    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" podStartSLOduration=2.282788896 podStartE2EDuration="48.405247061s" totalImagesPullingTime="39.605464794s" totalInitContainerRuntime="6.516993371s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:50:38.40383413 +0000 UTC m=+964.297834534" watchObservedRunningTime="2026-07-29 18:50:38.405247061 +0000 UTC m=+964.299247517"
Jul 29 18:50:40.982908 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:50:40.982874    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:50:40.983446 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:50:40.982955    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:50:40.984698 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:50:40.984669    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 18:50:50.983188 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:50:50.983138    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 18:51:00.983687 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:00.983588    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 18:51:10.983674 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:10.983628    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 18:51:20.983627 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:20.983581    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 18:51:27.697753 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:27.697713    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:51:30.983789 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:30.983738    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 18:51:35.046858 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.046822    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"]
Jul 29 18:51:35.049339 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.049319    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.052781 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.052755    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
Jul 29 18:51:35.064495 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.064467    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"]
Jul 29 18:51:35.075519 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.075485    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:51:35.206131 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.206089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e1784b-88da-4dda-966d-b7ee5b3de00f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.206332 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.206140    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.206332 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.206174    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.206332 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.206213    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.206332 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.206289    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.206540 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.206367    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wgsk\" (UniqueName: \"kubernetes.io/projected/98e1784b-88da-4dda-966d-b7ee5b3de00f-kube-api-access-4wgsk\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.206540 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.206405    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.307633 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.307601    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.307815 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.307655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.307815 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.307702    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.307815 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.307742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.307815 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.307794    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wgsk\" (UniqueName: \"kubernetes.io/projected/98e1784b-88da-4dda-966d-b7ee5b3de00f-kube-api-access-4wgsk\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.308036 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.307830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.308036 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.307895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e1784b-88da-4dda-966d-b7ee5b3de00f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.308519 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.308491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.308519 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.308504    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.309401 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.309360    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.309518 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.309500    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.310597 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.310569    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.310788 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.310758    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e1784b-88da-4dda-966d-b7ee5b3de00f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.316885 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.316859    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"]
Jul 29 18:51:35.320949 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.320926    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.322095 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.322066    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wgsk\" (UniqueName: \"kubernetes.io/projected/98e1784b-88da-4dda-966d-b7ee5b3de00f-kube-api-access-4wgsk\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.323702 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.323680    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa-dockercfg-8r422\""
Jul 29 18:51:35.333309 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.333284    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"]
Jul 29 18:51:35.359795 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.359764    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:35.382725 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.382692    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:51:35.409400 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.409361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.409651 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.409630    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.409842 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.409825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f055cc9-ad35-4873-9b52-1e138a9ef023-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.409970 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.409953    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.410103 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.410087    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnm4x\" (UniqueName: \"kubernetes.io/projected/0f055cc9-ad35-4873-9b52-1e138a9ef023-kube-api-access-bnm4x\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.410248 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.410232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.509132 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.509096    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"]
Jul 29 18:51:35.509856 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:51:35.509822    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e1784b_88da_4dda_966d_b7ee5b3de00f.slice/crio-d2a28947b82caab29d9522868ae0afe821b0742423ac95a2978df58712610619 WatchSource:0}: Error finding container d2a28947b82caab29d9522868ae0afe821b0742423ac95a2978df58712610619: Status 404 returned error can't find the container with id d2a28947b82caab29d9522868ae0afe821b0742423ac95a2978df58712610619
Jul 29 18:51:35.511107 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.511077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.511234 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.511158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.511234 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.511191    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.511364 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.511277    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f055cc9-ad35-4873-9b52-1e138a9ef023-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.511364 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.511307    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.511364 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.511353    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnm4x\" (UniqueName: \"kubernetes.io/projected/0f055cc9-ad35-4873-9b52-1e138a9ef023-kube-api-access-bnm4x\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.511931 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.511520    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.511931 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.511833    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.511931 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.511869    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.512136 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.511974    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.514819 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.514736    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f055cc9-ad35-4873-9b52-1e138a9ef023-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.522833 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.522806    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnm4x\" (UniqueName: \"kubernetes.io/projected/0f055cc9-ad35-4873-9b52-1e138a9ef023-kube-api-access-bnm4x\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.645543 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.645506    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:35.694579 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.694540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" event={"ID":"98e1784b-88da-4dda-966d-b7ee5b3de00f","Type":"ContainerStarted","Data":"4bd91cef42b72af9d495818faf32d30726b1abd42eee4d61d658e1ca4f55478f"}
Jul 29 18:51:35.694790 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.694610    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" event={"ID":"98e1784b-88da-4dda-966d-b7ee5b3de00f","Type":"ContainerStarted","Data":"d2a28947b82caab29d9522868ae0afe821b0742423ac95a2978df58712610619"}
Jul 29 18:51:35.793386 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:35.793334    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"]
Jul 29 18:51:35.799806 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:51:35.799776    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f055cc9_ad35_4873_9b52_1e138a9ef023.slice/crio-1a4bf88f8cb0f0397027ae9435241c586541b97041ab5dfe1dbee291a5d7b12c WatchSource:0}: Error finding container 1a4bf88f8cb0f0397027ae9435241c586541b97041ab5dfe1dbee291a5d7b12c: Status 404 returned error can't find the container with id 1a4bf88f8cb0f0397027ae9435241c586541b97041ab5dfe1dbee291a5d7b12c
Jul 29 18:51:36.702791 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:36.702744    2560 generic.go:356] "Generic (PLEG): container finished" podID="0f055cc9-ad35-4873-9b52-1e138a9ef023" containerID="cb249842476191cf8b623c4a465536ea202060c4595140a5990c5516b15f317e" exitCode=0
Jul 29 18:51:36.703269 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:36.702828    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6" event={"ID":"0f055cc9-ad35-4873-9b52-1e138a9ef023","Type":"ContainerDied","Data":"cb249842476191cf8b623c4a465536ea202060c4595140a5990c5516b15f317e"}
Jul 29 18:51:36.703269 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:36.702900    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6" event={"ID":"0f055cc9-ad35-4873-9b52-1e138a9ef023","Type":"ContainerStarted","Data":"1a4bf88f8cb0f0397027ae9435241c586541b97041ab5dfe1dbee291a5d7b12c"}
Jul 29 18:51:37.714804 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:37.714759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6" event={"ID":"0f055cc9-ad35-4873-9b52-1e138a9ef023","Type":"ContainerStarted","Data":"4d97b9ca4243758b8fae9ae80032b0eef84c704b9c31bd37fc12269bc1e1a609"}
Jul 29 18:51:37.715329 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:37.714858    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6" event={"ID":"0f055cc9-ad35-4873-9b52-1e138a9ef023","Type":"ContainerStarted","Data":"766c51f1224b9e83bd692ea4771ca69005bec4e45dc76173a4704671880122bd"}
Jul 29 18:51:37.715329 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:37.714917    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:37.737174 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:37.737115    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6" podStartSLOduration=1.936065624 podStartE2EDuration="2.737097503s" totalImagesPullingTime="0s" totalInitContainerRuntime="801.031879ms" isStatefulPod=false podCreationTimestamp="2026-07-29 18:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:51:37.734461382 +0000 UTC m=+1023.628461823" watchObservedRunningTime="2026-07-29 18:51:37.737097503 +0000 UTC m=+1023.631097916"
Jul 29 18:51:40.983532 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:40.983488    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 18:51:42.750307 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:42.750271    2560 generic.go:356] "Generic (PLEG): container finished" podID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerID="4bd91cef42b72af9d495818faf32d30726b1abd42eee4d61d658e1ca4f55478f" exitCode=0
Jul 29 18:51:42.750708 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:42.750346    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" event={"ID":"98e1784b-88da-4dda-966d-b7ee5b3de00f","Type":"ContainerDied","Data":"4bd91cef42b72af9d495818faf32d30726b1abd42eee4d61d658e1ca4f55478f"}
Jul 29 18:51:43.759092 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:43.759049    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" event={"ID":"98e1784b-88da-4dda-966d-b7ee5b3de00f","Type":"ContainerStarted","Data":"36b004802c71f8c0baf8f8e88de429d0408a6c215d58aa68d47d0e6bc984914d"}
Jul 29 18:51:43.780142 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:43.780086    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" podStartSLOduration=2.268025356 podStartE2EDuration="8.780068546s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.51204319s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:51:43.778668044 +0000 UTC m=+1029.672668471" watchObservedRunningTime="2026-07-29 18:51:43.780068546 +0000 UTC m=+1029.674068947"
Jul 29 18:51:45.360637 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:45.360592    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:45.361071 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:45.360664    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:51:45.362470 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:45.362439    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 29 18:51:45.646179 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:45.646079    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:45.646179 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:45.646154    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:45.649217 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:45.649166    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:45.773061 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:45.773029    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:51:50.983640 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:50.983586    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 18:51:55.361183 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:55.361133    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 29 18:51:56.371737 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:51:56.371701    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:52:00.983780 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:00.983724    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 18:52:05.361017 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:05.360965    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 29 18:52:06.780469 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:06.780440    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:52:07.286987 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:07.286956    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:52:07.387666 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:07.387637    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:52:10.993266 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:10.993225    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:52:11.002503 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:11.002463    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:52:11.931416 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:11.931380    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"]
Jul 29 18:52:11.950820 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:11.950785    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:52:12.062925 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:52:12.062886    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs: secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 18:52:12.063359 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:52:12.062979    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs podName:6a78c984-72be-4b36-9d93-7aaffb889e46 nodeName:}" failed. No retries permitted until 2026-07-29 18:52:12.562956121 +0000 UTC m=+1058.456956521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs") pod "stop-feature-test-kserve-6df78cd5bb-dq962" (UID: "6a78c984-72be-4b36-9d93-7aaffb889e46") : secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 18:52:12.141435 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:12.141391    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:52:12.240812 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:12.240732    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:52:12.568358 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:52:12.568330    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs: secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 18:52:12.568521 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:52:12.568403    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs podName:6a78c984-72be-4b36-9d93-7aaffb889e46 nodeName:}" failed. No retries permitted until 2026-07-29 18:52:13.568386862 +0000 UTC m=+1059.462387249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs") pod "stop-feature-test-kserve-6df78cd5bb-dq962" (UID: "6a78c984-72be-4b36-9d93-7aaffb889e46") : secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 18:52:12.947402 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:12.947296    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="main" containerID="cri-o://e92aedd12afab8f55cc1c4d7f9f972eaa11164dacbe9db1f173cdd015dde17ba" gracePeriod=30
Jul 29 18:52:13.577715 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:52:13.577680    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs: secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 18:52:13.578219 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:52:13.577768    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs podName:6a78c984-72be-4b36-9d93-7aaffb889e46 nodeName:}" failed. No retries permitted until 2026-07-29 18:52:15.57774865 +0000 UTC m=+1061.471749035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs") pod "stop-feature-test-kserve-6df78cd5bb-dq962" (UID: "6a78c984-72be-4b36-9d93-7aaffb889e46") : secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 18:52:15.361140 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:15.361088    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 29 18:52:15.596711 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:52:15.596673    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs: secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 18:52:15.596877 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:52:15.596757    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs podName:6a78c984-72be-4b36-9d93-7aaffb889e46 nodeName:}" failed. No retries permitted until 2026-07-29 18:52:19.596736845 +0000 UTC m=+1065.490737230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs") pod "stop-feature-test-kserve-6df78cd5bb-dq962" (UID: "6a78c984-72be-4b36-9d93-7aaffb889e46") : secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 18:52:19.633010 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:52:19.632978    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs: secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 18:52:19.633425 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:52:19.633042    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs podName:6a78c984-72be-4b36-9d93-7aaffb889e46 nodeName:}" failed. No retries permitted until 2026-07-29 18:52:27.633028073 +0000 UTC m=+1073.527028452 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs") pod "stop-feature-test-kserve-6df78cd5bb-dq962" (UID: "6a78c984-72be-4b36-9d93-7aaffb889e46") : secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 18:52:25.361121 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:25.361077    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 29 18:52:27.706981 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:52:27.706894    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs: secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 18:52:27.706981 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:52:27.706979    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs podName:6a78c984-72be-4b36-9d93-7aaffb889e46 nodeName:}" failed. No retries permitted until 2026-07-29 18:52:43.706963368 +0000 UTC m=+1089.600963751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs") pod "stop-feature-test-kserve-6df78cd5bb-dq962" (UID: "6a78c984-72be-4b36-9d93-7aaffb889e46") : secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 18:52:29.329810 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.329773    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:52:29.378109 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.378069    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"]
Jul 29 18:52:29.382590 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.382569    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.391220 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.391178    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"]
Jul 29 18:52:29.454101 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.454067    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:52:29.522726 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.522687    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-dshm\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.522911 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.522739    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-home\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.522911 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.522827    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdwbv\" (UniqueName: \"kubernetes.io/projected/2e8c51da-ebae-465b-8ee6-a047cd78f53f-kube-api-access-qdwbv\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.523002 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.522904    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-tmp-dir\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.523041 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.523016    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-kserve-provision-location\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.523086 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.523050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8c51da-ebae-465b-8ee6-a047cd78f53f-tls-certs\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.523086 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.523066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-model-cache\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.623668 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.623586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-kserve-provision-location\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.623668 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.623632    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8c51da-ebae-465b-8ee6-a047cd78f53f-tls-certs\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.623668 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.623648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-model-cache\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.623668 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.623664    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-dshm\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.623980 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.623700    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-home\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.623980 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.623723    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdwbv\" (UniqueName: \"kubernetes.io/projected/2e8c51da-ebae-465b-8ee6-a047cd78f53f-kube-api-access-qdwbv\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.623980 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.623754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-tmp-dir\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.624142 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.624085    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-home\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.624222 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.624149    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-tmp-dir\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.624222 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.624149    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-model-cache\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.624326 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.624230    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-kserve-provision-location\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.626091 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.626071    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-dshm\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.626429 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.626413    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8c51da-ebae-465b-8ee6-a047cd78f53f-tls-certs\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.631380 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.631358    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdwbv\" (UniqueName: \"kubernetes.io/projected/2e8c51da-ebae-465b-8ee6-a047cd78f53f-kube-api-access-qdwbv\") pod \"stop-feature-test-kserve-6df78cd5bb-vptf7\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.695989 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.695958    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:29.810308 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.810269    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:52:29.849599 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:29.849572    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"]
Jul 29 18:52:29.851775 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:52:29.851750    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8c51da_ebae_465b_8ee6_a047cd78f53f.slice/crio-e4ff20ed94aa7011e7503c9046f9ebe57855b3d85e160e649ca6d94fc92de84a WatchSource:0}: Error finding container e4ff20ed94aa7011e7503c9046f9ebe57855b3d85e160e649ca6d94fc92de84a: Status 404 returned error can't find the container with id e4ff20ed94aa7011e7503c9046f9ebe57855b3d85e160e649ca6d94fc92de84a
Jul 29 18:52:30.056339 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:30.056302    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" event={"ID":"2e8c51da-ebae-465b-8ee6-a047cd78f53f","Type":"ContainerStarted","Data":"9d71793cfc99a6939115dbc087ccc4f2c1f49d5d2013a8facacb0494604f3814"}
Jul 29 18:52:30.056541 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:30.056355    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" event={"ID":"2e8c51da-ebae-465b-8ee6-a047cd78f53f","Type":"ContainerStarted","Data":"e4ff20ed94aa7011e7503c9046f9ebe57855b3d85e160e649ca6d94fc92de84a"}
Jul 29 18:52:35.361242 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:35.361178    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 29 18:52:36.092350 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:36.092311    2560 generic.go:356] "Generic (PLEG): container finished" podID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerID="9d71793cfc99a6939115dbc087ccc4f2c1f49d5d2013a8facacb0494604f3814" exitCode=0
Jul 29 18:52:36.092544 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:36.092384    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" event={"ID":"2e8c51da-ebae-465b-8ee6-a047cd78f53f","Type":"ContainerDied","Data":"9d71793cfc99a6939115dbc087ccc4f2c1f49d5d2013a8facacb0494604f3814"}
Jul 29 18:52:37.102515 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:37.102470    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" event={"ID":"2e8c51da-ebae-465b-8ee6-a047cd78f53f","Type":"ContainerStarted","Data":"2a923ddfd4cf48db714a39ec4b475948837a2515f194c747ccdd03152bca23ec"}
Jul 29 18:52:37.125557 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:37.125504    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" podStartSLOduration=2.740169883 podStartE2EDuration="8.125488276s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.385318393s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:52:37.122233171 +0000 UTC m=+1083.016233596" watchObservedRunningTime="2026-07-29 18:52:37.125488276 +0000 UTC m=+1083.019488677"
Jul 29 18:52:39.696528 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:39.696494    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:39.696941 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:39.696575    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:52:39.698221 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:39.698172    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 29 18:52:41.573385 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:41.573350    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:52:43.143431 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.143338    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6df78cd5bb-dq962_6a78c984-72be-4b36-9d93-7aaffb889e46/main/0.log"
Jul 29 18:52:43.143901 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.143773    2560 generic.go:356] "Generic (PLEG): container finished" podID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerID="e92aedd12afab8f55cc1c4d7f9f972eaa11164dacbe9db1f173cdd015dde17ba" exitCode=137
Jul 29 18:52:43.143901 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.143868    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" event={"ID":"6a78c984-72be-4b36-9d93-7aaffb889e46","Type":"ContainerDied","Data":"e92aedd12afab8f55cc1c4d7f9f972eaa11164dacbe9db1f173cdd015dde17ba"}
Jul 29 18:52:43.246319 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.246295    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6df78cd5bb-dq962_6a78c984-72be-4b36-9d93-7aaffb889e46/main/0.log"
Jul 29 18:52:43.271146 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.271087    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:52:43.349254 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.349221    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmn9b\" (UniqueName: \"kubernetes.io/projected/6a78c984-72be-4b36-9d93-7aaffb889e46-kube-api-access-fmn9b\") pod \"6a78c984-72be-4b36-9d93-7aaffb889e46\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") "
Jul 29 18:52:43.349429 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.349341    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-tmp-dir\") pod \"6a78c984-72be-4b36-9d93-7aaffb889e46\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") "
Jul 29 18:52:43.349429 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.349366    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-home\") pod \"6a78c984-72be-4b36-9d93-7aaffb889e46\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") "
Jul 29 18:52:43.349429 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.349420    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-kserve-provision-location\") pod \"6a78c984-72be-4b36-9d93-7aaffb889e46\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") "
Jul 29 18:52:43.349635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.349455    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-dshm\") pod \"6a78c984-72be-4b36-9d93-7aaffb889e46\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") "
Jul 29 18:52:43.349635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.349494    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs\") pod \"6a78c984-72be-4b36-9d93-7aaffb889e46\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") "
Jul 29 18:52:43.349635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.349529    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-model-cache\") pod \"6a78c984-72be-4b36-9d93-7aaffb889e46\" (UID: \"6a78c984-72be-4b36-9d93-7aaffb889e46\") "
Jul 29 18:52:43.349982 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.349956    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-home" (OuterVolumeSpecName: "home") pod "6a78c984-72be-4b36-9d93-7aaffb889e46" (UID: "6a78c984-72be-4b36-9d93-7aaffb889e46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:52:43.350085 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.349966    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a78c984-72be-4b36-9d93-7aaffb889e46" (UID: "6a78c984-72be-4b36-9d93-7aaffb889e46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:52:43.351906 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.351864    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a78c984-72be-4b36-9d93-7aaffb889e46-kube-api-access-fmn9b" (OuterVolumeSpecName: "kube-api-access-fmn9b") pod "6a78c984-72be-4b36-9d93-7aaffb889e46" (UID: "6a78c984-72be-4b36-9d93-7aaffb889e46"). InnerVolumeSpecName "kube-api-access-fmn9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:52:43.352294 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.352267    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-dshm" (OuterVolumeSpecName: "dshm") pod "6a78c984-72be-4b36-9d93-7aaffb889e46" (UID: "6a78c984-72be-4b36-9d93-7aaffb889e46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:52:43.363075 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.363009    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a78c984-72be-4b36-9d93-7aaffb889e46" (UID: "6a78c984-72be-4b36-9d93-7aaffb889e46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:52:43.367676 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.367643    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a78c984-72be-4b36-9d93-7aaffb889e46" (UID: "6a78c984-72be-4b36-9d93-7aaffb889e46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:52:43.419474 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.419434    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a78c984-72be-4b36-9d93-7aaffb889e46" (UID: "6a78c984-72be-4b36-9d93-7aaffb889e46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:52:43.450833 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.450792    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:52:43.450833 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.450827    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:52:43.450833 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.450838    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a78c984-72be-4b36-9d93-7aaffb889e46-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:52:43.451137 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.450849    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:52:43.451137 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.450863    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmn9b\" (UniqueName: \"kubernetes.io/projected/6a78c984-72be-4b36-9d93-7aaffb889e46-kube-api-access-fmn9b\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:52:43.451137 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.450873    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:52:43.451137 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:43.450883    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a78c984-72be-4b36-9d93-7aaffb889e46-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:52:44.150517 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:44.150477    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6df78cd5bb-dq962_6a78c984-72be-4b36-9d93-7aaffb889e46/main/0.log"
Jul 29 18:52:44.151077 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:44.151051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962" event={"ID":"6a78c984-72be-4b36-9d93-7aaffb889e46","Type":"ContainerDied","Data":"3ccdbbf0b9b13cc385ce922a746754e9a13f503e7a3e0f012be46963e3a5c4e1"}
Jul 29 18:52:44.151157 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:44.151143    2560 scope.go:117] "RemoveContainer" containerID="e92aedd12afab8f55cc1c4d7f9f972eaa11164dacbe9db1f173cdd015dde17ba"
Jul 29 18:52:44.151263 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:44.151059    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"
Jul 29 18:52:44.161351 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:44.161331    2560 scope.go:117] "RemoveContainer" containerID="2abcad60ac46d4298b1c05e9e423e6b247b71ffb8065b374603cfa90c5dac73e"
Jul 29 18:52:44.179430 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:44.179401    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"]
Jul 29 18:52:44.183365 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:44.183342    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-dq962"]
Jul 29 18:52:44.689848 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:44.689816    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" path="/var/lib/kubelet/pods/6a78c984-72be-4b36-9d93-7aaffb889e46/volumes"
Jul 29 18:52:45.361007 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:45.360959    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 29 18:52:49.697308 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:49.697257    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 29 18:52:50.866852 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:50.866815    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:52:55.360624 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:55.360569    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 29 18:52:59.696931 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:52:59.696874    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 29 18:53:05.360720 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:05.360665    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 29 18:53:09.696825 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:09.696784    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 29 18:53:15.360923 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:15.360864    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 29 18:53:19.697076 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:19.697025    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 29 18:53:25.370185 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:25.370144    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:53:25.378164 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:25.378136    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:53:29.697002 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:29.696961    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 29 18:53:32.268260 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:32.268221    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"]
Jul 29 18:53:32.268713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:32.268530    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="main" containerID="cri-o://36b004802c71f8c0baf8f8e88de429d0408a6c215d58aa68d47d0e6bc984914d" gracePeriod=30
Jul 29 18:53:32.273493 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:32.273464    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:53:32.277853 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:32.277826    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"]
Jul 29 18:53:32.278117 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:32.278089    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6" podUID="0f055cc9-ad35-4873-9b52-1e138a9ef023" containerName="main" containerID="cri-o://766c51f1224b9e83bd692ea4771ca69005bec4e45dc76173a4704671880122bd" gracePeriod=30
Jul 29 18:53:32.278362 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:32.278115    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6" podUID="0f055cc9-ad35-4873-9b52-1e138a9ef023" containerName="tokenizer" containerID="cri-o://4d97b9ca4243758b8fae9ae80032b0eef84c704b9c31bd37fc12269bc1e1a609" gracePeriod=30
Jul 29 18:53:32.334467 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:32.334427    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:53:32.454349 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:32.454318    2560 generic.go:356] "Generic (PLEG): container finished" podID="0f055cc9-ad35-4873-9b52-1e138a9ef023" containerID="766c51f1224b9e83bd692ea4771ca69005bec4e45dc76173a4704671880122bd" exitCode=0
Jul 29 18:53:32.454532 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:32.454394    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6" event={"ID":"0f055cc9-ad35-4873-9b52-1e138a9ef023","Type":"ContainerDied","Data":"766c51f1224b9e83bd692ea4771ca69005bec4e45dc76173a4704671880122bd"}
Jul 29 18:53:33.650983 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.650954    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:53:33.727540 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.727448    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-kserve-provision-location\") pod \"0f055cc9-ad35-4873-9b52-1e138a9ef023\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") "
Jul 29 18:53:33.727540 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.727500    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-cache\") pod \"0f055cc9-ad35-4873-9b52-1e138a9ef023\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") "
Jul 29 18:53:33.727540 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.727527    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f055cc9-ad35-4873-9b52-1e138a9ef023-tls-certs\") pod \"0f055cc9-ad35-4873-9b52-1e138a9ef023\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") "
Jul 29 18:53:33.727824 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.727570    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-tmp\") pod \"0f055cc9-ad35-4873-9b52-1e138a9ef023\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") "
Jul 29 18:53:33.727824 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.727590    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-uds\") pod \"0f055cc9-ad35-4873-9b52-1e138a9ef023\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") "
Jul 29 18:53:33.727824 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.727610    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnm4x\" (UniqueName: \"kubernetes.io/projected/0f055cc9-ad35-4873-9b52-1e138a9ef023-kube-api-access-bnm4x\") pod \"0f055cc9-ad35-4873-9b52-1e138a9ef023\" (UID: \"0f055cc9-ad35-4873-9b52-1e138a9ef023\") "
Jul 29 18:53:33.727824 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.727790    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "0f055cc9-ad35-4873-9b52-1e138a9ef023" (UID: "0f055cc9-ad35-4873-9b52-1e138a9ef023"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:53:33.727824 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.727803    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0f055cc9-ad35-4873-9b52-1e138a9ef023" (UID: "0f055cc9-ad35-4873-9b52-1e138a9ef023"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:53:33.728087 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.727873    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:53:33.728087 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.727890    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-uds\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:53:33.728087 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.727895    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "0f055cc9-ad35-4873-9b52-1e138a9ef023" (UID: "0f055cc9-ad35-4873-9b52-1e138a9ef023"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:53:33.728263 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.728159    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f055cc9-ad35-4873-9b52-1e138a9ef023" (UID: "0f055cc9-ad35-4873-9b52-1e138a9ef023"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:53:33.729705 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.729681    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f055cc9-ad35-4873-9b52-1e138a9ef023-kube-api-access-bnm4x" (OuterVolumeSpecName: "kube-api-access-bnm4x") pod "0f055cc9-ad35-4873-9b52-1e138a9ef023" (UID: "0f055cc9-ad35-4873-9b52-1e138a9ef023"). InnerVolumeSpecName "kube-api-access-bnm4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:53:33.729873 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.729848    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f055cc9-ad35-4873-9b52-1e138a9ef023-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f055cc9-ad35-4873-9b52-1e138a9ef023" (UID: "0f055cc9-ad35-4873-9b52-1e138a9ef023"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:53:33.828583 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.828545    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:53:33.828583 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.828573    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f055cc9-ad35-4873-9b52-1e138a9ef023-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:53:33.828583 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.828585    2560 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f055cc9-ad35-4873-9b52-1e138a9ef023-tokenizer-tmp\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:53:33.828583 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:33.828593    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnm4x\" (UniqueName: \"kubernetes.io/projected/0f055cc9-ad35-4873-9b52-1e138a9ef023-kube-api-access-bnm4x\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:53:34.468609 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.468578    2560 generic.go:356] "Generic (PLEG): container finished" podID="0f055cc9-ad35-4873-9b52-1e138a9ef023" containerID="4d97b9ca4243758b8fae9ae80032b0eef84c704b9c31bd37fc12269bc1e1a609" exitCode=0
Jul 29 18:53:34.468812 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.468633    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"
Jul 29 18:53:34.468812 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.468665    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6" event={"ID":"0f055cc9-ad35-4873-9b52-1e138a9ef023","Type":"ContainerDied","Data":"4d97b9ca4243758b8fae9ae80032b0eef84c704b9c31bd37fc12269bc1e1a609"}
Jul 29 18:53:34.468812 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.468738    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6" event={"ID":"0f055cc9-ad35-4873-9b52-1e138a9ef023","Type":"ContainerDied","Data":"1a4bf88f8cb0f0397027ae9435241c586541b97041ab5dfe1dbee291a5d7b12c"}
Jul 29 18:53:34.468812 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.468773    2560 scope.go:117] "RemoveContainer" containerID="4d97b9ca4243758b8fae9ae80032b0eef84c704b9c31bd37fc12269bc1e1a609"
Jul 29 18:53:34.478422 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.478338    2560 scope.go:117] "RemoveContainer" containerID="766c51f1224b9e83bd692ea4771ca69005bec4e45dc76173a4704671880122bd"
Jul 29 18:53:34.487081 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.487056    2560 scope.go:117] "RemoveContainer" containerID="cb249842476191cf8b623c4a465536ea202060c4595140a5990c5516b15f317e"
Jul 29 18:53:34.492880 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.492855    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"]
Jul 29 18:53:34.497228 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.497030    2560 scope.go:117] "RemoveContainer" containerID="4d97b9ca4243758b8fae9ae80032b0eef84c704b9c31bd37fc12269bc1e1a609"
Jul 29 18:53:34.497414 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:53:34.497391    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d97b9ca4243758b8fae9ae80032b0eef84c704b9c31bd37fc12269bc1e1a609\": container with ID starting with 4d97b9ca4243758b8fae9ae80032b0eef84c704b9c31bd37fc12269bc1e1a609 not found: ID does not exist" containerID="4d97b9ca4243758b8fae9ae80032b0eef84c704b9c31bd37fc12269bc1e1a609"
Jul 29 18:53:34.497473 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.497423    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d97b9ca4243758b8fae9ae80032b0eef84c704b9c31bd37fc12269bc1e1a609"} err="failed to get container status \"4d97b9ca4243758b8fae9ae80032b0eef84c704b9c31bd37fc12269bc1e1a609\": rpc error: code = NotFound desc = could not find container \"4d97b9ca4243758b8fae9ae80032b0eef84c704b9c31bd37fc12269bc1e1a609\": container with ID starting with 4d97b9ca4243758b8fae9ae80032b0eef84c704b9c31bd37fc12269bc1e1a609 not found: ID does not exist"
Jul 29 18:53:34.497473 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.497441    2560 scope.go:117] "RemoveContainer" containerID="766c51f1224b9e83bd692ea4771ca69005bec4e45dc76173a4704671880122bd"
Jul 29 18:53:34.497722 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:53:34.497701    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"766c51f1224b9e83bd692ea4771ca69005bec4e45dc76173a4704671880122bd\": container with ID starting with 766c51f1224b9e83bd692ea4771ca69005bec4e45dc76173a4704671880122bd not found: ID does not exist" containerID="766c51f1224b9e83bd692ea4771ca69005bec4e45dc76173a4704671880122bd"
Jul 29 18:53:34.497786 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.497727    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"766c51f1224b9e83bd692ea4771ca69005bec4e45dc76173a4704671880122bd"} err="failed to get container status \"766c51f1224b9e83bd692ea4771ca69005bec4e45dc76173a4704671880122bd\": rpc error: code = NotFound desc = could not find container \"766c51f1224b9e83bd692ea4771ca69005bec4e45dc76173a4704671880122bd\": container with ID starting with 766c51f1224b9e83bd692ea4771ca69005bec4e45dc76173a4704671880122bd not found: ID does not exist"
Jul 29 18:53:34.497786 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.497745    2560 scope.go:117] "RemoveContainer" containerID="cb249842476191cf8b623c4a465536ea202060c4595140a5990c5516b15f317e"
Jul 29 18:53:34.498009 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:53:34.497992    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb249842476191cf8b623c4a465536ea202060c4595140a5990c5516b15f317e\": container with ID starting with cb249842476191cf8b623c4a465536ea202060c4595140a5990c5516b15f317e not found: ID does not exist" containerID="cb249842476191cf8b623c4a465536ea202060c4595140a5990c5516b15f317e"
Jul 29 18:53:34.498056 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.498014    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb249842476191cf8b623c4a465536ea202060c4595140a5990c5516b15f317e"} err="failed to get container status \"cb249842476191cf8b623c4a465536ea202060c4595140a5990c5516b15f317e\": rpc error: code = NotFound desc = could not find container \"cb249842476191cf8b623c4a465536ea202060c4595140a5990c5516b15f317e\": container with ID starting with cb249842476191cf8b623c4a465536ea202060c4595140a5990c5516b15f317e not found: ID does not exist"
Jul 29 18:53:34.498107 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.498074    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schetnkb6"]
Jul 29 18:53:34.693665 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:34.689218    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f055cc9-ad35-4873-9b52-1e138a9ef023" path="/var/lib/kubelet/pods/0f055cc9-ad35-4873-9b52-1e138a9ef023/volumes"
Jul 29 18:53:37.565829 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.565795    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"]
Jul 29 18:53:37.566224 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.566149    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f055cc9-ad35-4873-9b52-1e138a9ef023" containerName="tokenizer"
Jul 29 18:53:37.566224 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.566160    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f055cc9-ad35-4873-9b52-1e138a9ef023" containerName="tokenizer"
Jul 29 18:53:37.566224 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.566171    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f055cc9-ad35-4873-9b52-1e138a9ef023" containerName="storage-initializer"
Jul 29 18:53:37.566224 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.566177    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f055cc9-ad35-4873-9b52-1e138a9ef023" containerName="storage-initializer"
Jul 29 18:53:37.566224 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.566191    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f055cc9-ad35-4873-9b52-1e138a9ef023" containerName="main"
Jul 29 18:53:37.566224 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.566224    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f055cc9-ad35-4873-9b52-1e138a9ef023" containerName="main"
Jul 29 18:53:37.566430 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.566234    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="main"
Jul 29 18:53:37.566430 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.566241    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="main"
Jul 29 18:53:37.566430 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.566252    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="storage-initializer"
Jul 29 18:53:37.566430 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.566257    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="storage-initializer"
Jul 29 18:53:37.566430 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.566317    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a78c984-72be-4b36-9d93-7aaffb889e46" containerName="main"
Jul 29 18:53:37.566430 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.566324    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f055cc9-ad35-4873-9b52-1e138a9ef023" containerName="main"
Jul 29 18:53:37.566430 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.566330    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f055cc9-ad35-4873-9b52-1e138a9ef023" containerName="tokenizer"
Jul 29 18:53:37.569957 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.569939    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.572529 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.572508    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 29 18:53:37.580870 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.580842    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"]
Jul 29 18:53:37.593038 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.593000    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:53:37.664317 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.664218    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-dshm\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.664317 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.664267    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-tmp-dir\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.664317 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.664300    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-home\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.664317 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.664317    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.664671 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.664411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-model-cache\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.664671 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.664445    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/128383ad-4b37-4851-adfa-149221d186cc-tls-certs\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.664671 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.664484    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5vfq\" (UniqueName: \"kubernetes.io/projected/128383ad-4b37-4851-adfa-149221d186cc-kube-api-access-n5vfq\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.765598 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.765547    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-tmp-dir\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.765598 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.765594    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-home\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.765862 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.765611    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.765862 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.765639    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-model-cache\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.765862 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.765667    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/128383ad-4b37-4851-adfa-149221d186cc-tls-certs\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.765862 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.765683    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5vfq\" (UniqueName: \"kubernetes.io/projected/128383ad-4b37-4851-adfa-149221d186cc-kube-api-access-n5vfq\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.765862 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.765742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-dshm\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.766167 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.766005    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-home\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.766167 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.766043    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-model-cache\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.766167 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.766066    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-tmp-dir\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.766167 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.766128    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.767963 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.767938    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-dshm\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.768261 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.768238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/128383ad-4b37-4851-adfa-149221d186cc-tls-certs\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.775863 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.775836    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5vfq\" (UniqueName: \"kubernetes.io/projected/128383ad-4b37-4851-adfa-149221d186cc-kube-api-access-n5vfq\") pod \"custom-route-timeout-test-kserve-c7b9f676b-r2k65\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:37.856362 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.856326    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:53:37.883766 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:37.883739    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:38.031993 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:38.031969    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"]
Jul 29 18:53:38.035363 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:53:38.035326    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128383ad_4b37_4851_adfa_149221d186cc.slice/crio-bf5e6b14b889530b6a81644c9f6f39dba82d6ee88ebb65e5823a5b835daac190 WatchSource:0}: Error finding container bf5e6b14b889530b6a81644c9f6f39dba82d6ee88ebb65e5823a5b835daac190: Status 404 returned error can't find the container with id bf5e6b14b889530b6a81644c9f6f39dba82d6ee88ebb65e5823a5b835daac190
Jul 29 18:53:38.493814 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:38.493778    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" event={"ID":"128383ad-4b37-4851-adfa-149221d186cc","Type":"ContainerStarted","Data":"4c5c86328969e40d8a36a5ca5cf08aea0cd04ec48614b1d93ed1aa3df8f438eb"}
Jul 29 18:53:38.494031 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:38.493831    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" event={"ID":"128383ad-4b37-4851-adfa-149221d186cc","Type":"ContainerStarted","Data":"bf5e6b14b889530b6a81644c9f6f39dba82d6ee88ebb65e5823a5b835daac190"}
Jul 29 18:53:39.696727 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:39.696653    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 29 18:53:44.533686 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:44.533647    2560 generic.go:356] "Generic (PLEG): container finished" podID="128383ad-4b37-4851-adfa-149221d186cc" containerID="4c5c86328969e40d8a36a5ca5cf08aea0cd04ec48614b1d93ed1aa3df8f438eb" exitCode=0
Jul 29 18:53:44.534118 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:44.533718    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" event={"ID":"128383ad-4b37-4851-adfa-149221d186cc","Type":"ContainerDied","Data":"4c5c86328969e40d8a36a5ca5cf08aea0cd04ec48614b1d93ed1aa3df8f438eb"}
Jul 29 18:53:45.542602 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:45.542559    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" event={"ID":"128383ad-4b37-4851-adfa-149221d186cc","Type":"ContainerStarted","Data":"08c33b9a82f39f5a679dabfe8ba7af7e32dc4132951742187f5b4033fff185d7"}
Jul 29 18:53:45.568221 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:45.568151    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" podStartSLOduration=2.630054653 podStartE2EDuration="8.568134882s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.938080229s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:53:45.567298018 +0000 UTC m=+1151.461298446" watchObservedRunningTime="2026-07-29 18:53:45.568134882 +0000 UTC m=+1151.462135282"
Jul 29 18:53:47.884248 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:47.884205    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:47.884754 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:47.884296    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:53:47.885985 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:47.885954    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 29 18:53:49.696522 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:49.696462    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 29 18:53:50.708772 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:50.708740    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:53:57.884827 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:57.884727    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 29 18:53:59.697003 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:53:59.696949    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 29 18:54:02.579842 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.579806    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s_98e1784b-88da-4dda-966d-b7ee5b3de00f/main/0.log"
Jul 29 18:54:02.581896 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.581878    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:54:02.599998 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.599967    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-home\") pod \"98e1784b-88da-4dda-966d-b7ee5b3de00f\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") "
Jul 29 18:54:02.600131 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.600023    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e1784b-88da-4dda-966d-b7ee5b3de00f-tls-certs\") pod \"98e1784b-88da-4dda-966d-b7ee5b3de00f\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") "
Jul 29 18:54:02.600131 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.600081    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-model-cache\") pod \"98e1784b-88da-4dda-966d-b7ee5b3de00f\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") "
Jul 29 18:54:02.600131 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.600111    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-kserve-provision-location\") pod \"98e1784b-88da-4dda-966d-b7ee5b3de00f\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") "
Jul 29 18:54:02.600364 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.600160    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-dshm\") pod \"98e1784b-88da-4dda-966d-b7ee5b3de00f\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") "
Jul 29 18:54:02.600364 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.600228    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wgsk\" (UniqueName: \"kubernetes.io/projected/98e1784b-88da-4dda-966d-b7ee5b3de00f-kube-api-access-4wgsk\") pod \"98e1784b-88da-4dda-966d-b7ee5b3de00f\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") "
Jul 29 18:54:02.600364 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.600290    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-tmp-dir\") pod \"98e1784b-88da-4dda-966d-b7ee5b3de00f\" (UID: \"98e1784b-88da-4dda-966d-b7ee5b3de00f\") "
Jul 29 18:54:02.600521 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.600390    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-model-cache" (OuterVolumeSpecName: "model-cache") pod "98e1784b-88da-4dda-966d-b7ee5b3de00f" (UID: "98e1784b-88da-4dda-966d-b7ee5b3de00f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:54:02.600586 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.600563    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:02.600846 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.600818    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-home" (OuterVolumeSpecName: "home") pod "98e1784b-88da-4dda-966d-b7ee5b3de00f" (UID: "98e1784b-88da-4dda-966d-b7ee5b3de00f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:54:02.605117 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.605077    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-dshm" (OuterVolumeSpecName: "dshm") pod "98e1784b-88da-4dda-966d-b7ee5b3de00f" (UID: "98e1784b-88da-4dda-966d-b7ee5b3de00f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:54:02.605117 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.605089    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e1784b-88da-4dda-966d-b7ee5b3de00f-kube-api-access-4wgsk" (OuterVolumeSpecName: "kube-api-access-4wgsk") pod "98e1784b-88da-4dda-966d-b7ee5b3de00f" (UID: "98e1784b-88da-4dda-966d-b7ee5b3de00f"). InnerVolumeSpecName "kube-api-access-4wgsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:54:02.605117 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.605077    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98e1784b-88da-4dda-966d-b7ee5b3de00f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98e1784b-88da-4dda-966d-b7ee5b3de00f" (UID: "98e1784b-88da-4dda-966d-b7ee5b3de00f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:54:02.619463 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.619428    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98e1784b-88da-4dda-966d-b7ee5b3de00f" (UID: "98e1784b-88da-4dda-966d-b7ee5b3de00f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:54:02.647291 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.647261    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s_98e1784b-88da-4dda-966d-b7ee5b3de00f/main/0.log"
Jul 29 18:54:02.647732 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.647702    2560 generic.go:356] "Generic (PLEG): container finished" podID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerID="36b004802c71f8c0baf8f8e88de429d0408a6c215d58aa68d47d0e6bc984914d" exitCode=137
Jul 29 18:54:02.647886 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.647781    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"
Jul 29 18:54:02.647886 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.647799    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" event={"ID":"98e1784b-88da-4dda-966d-b7ee5b3de00f","Type":"ContainerDied","Data":"36b004802c71f8c0baf8f8e88de429d0408a6c215d58aa68d47d0e6bc984914d"}
Jul 29 18:54:02.647999 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.647902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s" event={"ID":"98e1784b-88da-4dda-966d-b7ee5b3de00f","Type":"ContainerDied","Data":"d2a28947b82caab29d9522868ae0afe821b0742423ac95a2978df58712610619"}
Jul 29 18:54:02.647999 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.647934    2560 scope.go:117] "RemoveContainer" containerID="36b004802c71f8c0baf8f8e88de429d0408a6c215d58aa68d47d0e6bc984914d"
Jul 29 18:54:02.658414 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.658388    2560 scope.go:117] "RemoveContainer" containerID="4bd91cef42b72af9d495818faf32d30726b1abd42eee4d61d658e1ca4f55478f"
Jul 29 18:54:02.679994 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.679952    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98e1784b-88da-4dda-966d-b7ee5b3de00f" (UID: "98e1784b-88da-4dda-966d-b7ee5b3de00f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:54:02.700964 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.700940    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:02.700964 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.700963    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e1784b-88da-4dda-966d-b7ee5b3de00f-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:02.701115 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.700973    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:02.701115 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.700983    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:02.701115 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.700993    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wgsk\" (UniqueName: \"kubernetes.io/projected/98e1784b-88da-4dda-966d-b7ee5b3de00f-kube-api-access-4wgsk\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:02.701115 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.701003    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98e1784b-88da-4dda-966d-b7ee5b3de00f-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:02.718873 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.718824    2560 scope.go:117] "RemoveContainer" containerID="36b004802c71f8c0baf8f8e88de429d0408a6c215d58aa68d47d0e6bc984914d"
Jul 29 18:54:02.719243 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:54:02.719219    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b004802c71f8c0baf8f8e88de429d0408a6c215d58aa68d47d0e6bc984914d\": container with ID starting with 36b004802c71f8c0baf8f8e88de429d0408a6c215d58aa68d47d0e6bc984914d not found: ID does not exist" containerID="36b004802c71f8c0baf8f8e88de429d0408a6c215d58aa68d47d0e6bc984914d"
Jul 29 18:54:02.719344 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.719256    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b004802c71f8c0baf8f8e88de429d0408a6c215d58aa68d47d0e6bc984914d"} err="failed to get container status \"36b004802c71f8c0baf8f8e88de429d0408a6c215d58aa68d47d0e6bc984914d\": rpc error: code = NotFound desc = could not find container \"36b004802c71f8c0baf8f8e88de429d0408a6c215d58aa68d47d0e6bc984914d\": container with ID starting with 36b004802c71f8c0baf8f8e88de429d0408a6c215d58aa68d47d0e6bc984914d not found: ID does not exist"
Jul 29 18:54:02.719344 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.719279    2560 scope.go:117] "RemoveContainer" containerID="4bd91cef42b72af9d495818faf32d30726b1abd42eee4d61d658e1ca4f55478f"
Jul 29 18:54:02.719655 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:54:02.719630    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd91cef42b72af9d495818faf32d30726b1abd42eee4d61d658e1ca4f55478f\": container with ID starting with 4bd91cef42b72af9d495818faf32d30726b1abd42eee4d61d658e1ca4f55478f not found: ID does not exist" containerID="4bd91cef42b72af9d495818faf32d30726b1abd42eee4d61d658e1ca4f55478f"
Jul 29 18:54:02.719726 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.719659    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd91cef42b72af9d495818faf32d30726b1abd42eee4d61d658e1ca4f55478f"} err="failed to get container status \"4bd91cef42b72af9d495818faf32d30726b1abd42eee4d61d658e1ca4f55478f\": rpc error: code = NotFound desc = could not find container \"4bd91cef42b72af9d495818faf32d30726b1abd42eee4d61d658e1ca4f55478f\": container with ID starting with 4bd91cef42b72af9d495818faf32d30726b1abd42eee4d61d658e1ca4f55478f not found: ID does not exist"
Jul 29 18:54:02.970650 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.970519    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"]
Jul 29 18:54:02.977805 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:02.977757    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-669bd94d58zb86s"]
Jul 29 18:54:04.688682 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:04.688645    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" path="/var/lib/kubelet/pods/98e1784b-88da-4dda-966d-b7ee5b3de00f/volumes"
Jul 29 18:54:07.884552 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:07.884499    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 29 18:54:09.696668 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:09.696621    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 29 18:54:17.885061 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:17.885007    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 29 18:54:19.707605 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:19.707570    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:54:19.720296 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:19.720266    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:54:20.817670 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:20.817635    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"]
Jul 29 18:54:20.818214 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:20.818160    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="main" containerID="cri-o://2a923ddfd4cf48db714a39ec4b475948837a2515f194c747ccdd03152bca23ec" gracePeriod=30
Jul 29 18:54:20.822416 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:20.822382    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:54:20.881828 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:20.881794    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:54:23.166670 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.166633    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-78cc98c464-zhjs8"]
Jul 29 18:54:23.167065 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.167007    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="storage-initializer"
Jul 29 18:54:23.167065 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.167019    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="storage-initializer"
Jul 29 18:54:23.167065 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.167044    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="main"
Jul 29 18:54:23.167065 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.167050    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="main"
Jul 29 18:54:23.167228 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.167109    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="98e1784b-88da-4dda-966d-b7ee5b3de00f" containerName="main"
Jul 29 18:54:23.170040 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.170023    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-78cc98c464-zhjs8"
Jul 29 18:54:23.179837 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.179813    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-78cc98c464-zhjs8"]
Jul 29 18:54:23.286842 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.286789    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfk7w\" (UniqueName: \"kubernetes.io/projected/5c131702-2ddc-4aa7-a4e0-35580c76401a-kube-api-access-dfk7w\") pod \"llmisvc-controller-manager-78cc98c464-zhjs8\" (UID: \"5c131702-2ddc-4aa7-a4e0-35580c76401a\") " pod="kserve/llmisvc-controller-manager-78cc98c464-zhjs8"
Jul 29 18:54:23.287018 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.286920    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c131702-2ddc-4aa7-a4e0-35580c76401a-cert\") pod \"llmisvc-controller-manager-78cc98c464-zhjs8\" (UID: \"5c131702-2ddc-4aa7-a4e0-35580c76401a\") " pod="kserve/llmisvc-controller-manager-78cc98c464-zhjs8"
Jul 29 18:54:23.387977 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.387930    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfk7w\" (UniqueName: \"kubernetes.io/projected/5c131702-2ddc-4aa7-a4e0-35580c76401a-kube-api-access-dfk7w\") pod \"llmisvc-controller-manager-78cc98c464-zhjs8\" (UID: \"5c131702-2ddc-4aa7-a4e0-35580c76401a\") " pod="kserve/llmisvc-controller-manager-78cc98c464-zhjs8"
Jul 29 18:54:23.388183 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.388033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c131702-2ddc-4aa7-a4e0-35580c76401a-cert\") pod \"llmisvc-controller-manager-78cc98c464-zhjs8\" (UID: \"5c131702-2ddc-4aa7-a4e0-35580c76401a\") " pod="kserve/llmisvc-controller-manager-78cc98c464-zhjs8"
Jul 29 18:54:23.390323 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.390300    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c131702-2ddc-4aa7-a4e0-35580c76401a-cert\") pod \"llmisvc-controller-manager-78cc98c464-zhjs8\" (UID: \"5c131702-2ddc-4aa7-a4e0-35580c76401a\") " pod="kserve/llmisvc-controller-manager-78cc98c464-zhjs8"
Jul 29 18:54:23.397363 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.397335    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfk7w\" (UniqueName: \"kubernetes.io/projected/5c131702-2ddc-4aa7-a4e0-35580c76401a-kube-api-access-dfk7w\") pod \"llmisvc-controller-manager-78cc98c464-zhjs8\" (UID: \"5c131702-2ddc-4aa7-a4e0-35580c76401a\") " pod="kserve/llmisvc-controller-manager-78cc98c464-zhjs8"
Jul 29 18:54:23.480578 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.480487    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-78cc98c464-zhjs8"
Jul 29 18:54:23.618472 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.618440    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-78cc98c464-zhjs8"]
Jul 29 18:54:23.619967 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:54:23.619930    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5c131702_2ddc_4aa7_a4e0_35580c76401a.slice/crio-f794295b9af083bd7725327ee1e8f18bd53cd2fb2c60e91eb4d618de0a939c0f WatchSource:0}: Error finding container f794295b9af083bd7725327ee1e8f18bd53cd2fb2c60e91eb4d618de0a939c0f: Status 404 returned error can't find the container with id f794295b9af083bd7725327ee1e8f18bd53cd2fb2c60e91eb4d618de0a939c0f
Jul 29 18:54:23.621817 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.621794    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 18:54:23.767065 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:23.766979    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-78cc98c464-zhjs8" event={"ID":"5c131702-2ddc-4aa7-a4e0-35580c76401a","Type":"ContainerStarted","Data":"f794295b9af083bd7725327ee1e8f18bd53cd2fb2c60e91eb4d618de0a939c0f"}
Jul 29 18:54:24.774949 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:24.774913    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-78cc98c464-zhjs8" event={"ID":"5c131702-2ddc-4aa7-a4e0-35580c76401a","Type":"ContainerStarted","Data":"06c7c99ca5ebff898b3b4323efbd4d4f3be7a8cf83e8169082c4161a70a8d6cb"}
Jul 29 18:54:24.775455 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:24.774998    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-78cc98c464-zhjs8"
Jul 29 18:54:24.792562 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:24.792507    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-78cc98c464-zhjs8" podStartSLOduration=1.123151325 podStartE2EDuration="1.792492312s" totalImagesPullingTime="669.340987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:54:24.791855737 +0000 UTC m=+1190.685856137" watchObservedRunningTime="2026-07-29 18:54:24.792492312 +0000 UTC m=+1190.686492713"
Jul 29 18:54:27.885286 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:27.885236    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 29 18:54:34.565164 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:34.565132    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 18:54:34.567806 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:34.567786    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 18:54:37.884655 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:37.884612    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 29 18:54:47.884881 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:47.884823    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 29 18:54:51.240928 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.240904    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6df78cd5bb-vptf7_2e8c51da-ebae-465b-8ee6-a047cd78f53f/main/0.log"
Jul 29 18:54:51.243890 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.243868    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:54:51.336688 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.336656    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-dshm\") pod \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") "
Jul 29 18:54:51.336688 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.336690    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-home\") pod \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") "
Jul 29 18:54:51.336944 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.336732    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-tmp-dir\") pod \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") "
Jul 29 18:54:51.336944 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.336773    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-kserve-provision-location\") pod \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") "
Jul 29 18:54:51.336944 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.336793    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-model-cache\") pod \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") "
Jul 29 18:54:51.336944 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.336832    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdwbv\" (UniqueName: \"kubernetes.io/projected/2e8c51da-ebae-465b-8ee6-a047cd78f53f-kube-api-access-qdwbv\") pod \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") "
Jul 29 18:54:51.336944 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.336856    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8c51da-ebae-465b-8ee6-a047cd78f53f-tls-certs\") pod \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\" (UID: \"2e8c51da-ebae-465b-8ee6-a047cd78f53f\") "
Jul 29 18:54:51.337247 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.337167    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e8c51da-ebae-465b-8ee6-a047cd78f53f" (UID: "2e8c51da-ebae-465b-8ee6-a047cd78f53f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:54:51.337460 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.337438    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-home" (OuterVolumeSpecName: "home") pod "2e8c51da-ebae-465b-8ee6-a047cd78f53f" (UID: "2e8c51da-ebae-465b-8ee6-a047cd78f53f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:54:51.339098 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.339014    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-dshm" (OuterVolumeSpecName: "dshm") pod "2e8c51da-ebae-465b-8ee6-a047cd78f53f" (UID: "2e8c51da-ebae-465b-8ee6-a047cd78f53f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:54:51.339254 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.339153    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8c51da-ebae-465b-8ee6-a047cd78f53f-kube-api-access-qdwbv" (OuterVolumeSpecName: "kube-api-access-qdwbv") pod "2e8c51da-ebae-465b-8ee6-a047cd78f53f" (UID: "2e8c51da-ebae-465b-8ee6-a047cd78f53f"). InnerVolumeSpecName "kube-api-access-qdwbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:54:51.339618 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.339588    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e8c51da-ebae-465b-8ee6-a047cd78f53f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e8c51da-ebae-465b-8ee6-a047cd78f53f" (UID: "2e8c51da-ebae-465b-8ee6-a047cd78f53f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:54:51.349385 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.349364    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e8c51da-ebae-465b-8ee6-a047cd78f53f" (UID: "2e8c51da-ebae-465b-8ee6-a047cd78f53f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:54:51.405044 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.404983    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e8c51da-ebae-465b-8ee6-a047cd78f53f" (UID: "2e8c51da-ebae-465b-8ee6-a047cd78f53f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:54:51.438211 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.438164    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:51.438211 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.438191    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdwbv\" (UniqueName: \"kubernetes.io/projected/2e8c51da-ebae-465b-8ee6-a047cd78f53f-kube-api-access-qdwbv\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:51.438400 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.438222    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8c51da-ebae-465b-8ee6-a047cd78f53f-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:51.438400 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.438234    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:51.438400 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.438241    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:51.438400 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.438249    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:51.438400 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.438256    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e8c51da-ebae-465b-8ee6-a047cd78f53f-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:51.926189 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.926144    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6df78cd5bb-vptf7_2e8c51da-ebae-465b-8ee6-a047cd78f53f/main/0.log"
Jul 29 18:54:51.929683 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.929645    2560 generic.go:356] "Generic (PLEG): container finished" podID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerID="2a923ddfd4cf48db714a39ec4b475948837a2515f194c747ccdd03152bca23ec" exitCode=137
Jul 29 18:54:51.929839 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.929740    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" event={"ID":"2e8c51da-ebae-465b-8ee6-a047cd78f53f","Type":"ContainerDied","Data":"2a923ddfd4cf48db714a39ec4b475948837a2515f194c747ccdd03152bca23ec"}
Jul 29 18:54:51.929839 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.929755    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"
Jul 29 18:54:51.929839 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.929794    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7" event={"ID":"2e8c51da-ebae-465b-8ee6-a047cd78f53f","Type":"ContainerDied","Data":"e4ff20ed94aa7011e7503c9046f9ebe57855b3d85e160e649ca6d94fc92de84a"}
Jul 29 18:54:51.929839 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.929824    2560 scope.go:117] "RemoveContainer" containerID="2a923ddfd4cf48db714a39ec4b475948837a2515f194c747ccdd03152bca23ec"
Jul 29 18:54:51.940027 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.940008    2560 scope.go:117] "RemoveContainer" containerID="9d71793cfc99a6939115dbc087ccc4f2c1f49d5d2013a8facacb0494604f3814"
Jul 29 18:54:51.950382 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.950360    2560 scope.go:117] "RemoveContainer" containerID="2a923ddfd4cf48db714a39ec4b475948837a2515f194c747ccdd03152bca23ec"
Jul 29 18:54:51.950659 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:54:51.950639    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a923ddfd4cf48db714a39ec4b475948837a2515f194c747ccdd03152bca23ec\": container with ID starting with 2a923ddfd4cf48db714a39ec4b475948837a2515f194c747ccdd03152bca23ec not found: ID does not exist" containerID="2a923ddfd4cf48db714a39ec4b475948837a2515f194c747ccdd03152bca23ec"
Jul 29 18:54:51.950709 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.950669    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a923ddfd4cf48db714a39ec4b475948837a2515f194c747ccdd03152bca23ec"} err="failed to get container status \"2a923ddfd4cf48db714a39ec4b475948837a2515f194c747ccdd03152bca23ec\": rpc error: code = NotFound desc = could not find container \"2a923ddfd4cf48db714a39ec4b475948837a2515f194c747ccdd03152bca23ec\": container with ID starting with 2a923ddfd4cf48db714a39ec4b475948837a2515f194c747ccdd03152bca23ec not found: ID does not exist"
Jul 29 18:54:51.950709 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.950686    2560 scope.go:117] "RemoveContainer" containerID="9d71793cfc99a6939115dbc087ccc4f2c1f49d5d2013a8facacb0494604f3814"
Jul 29 18:54:51.950942 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:54:51.950920    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d71793cfc99a6939115dbc087ccc4f2c1f49d5d2013a8facacb0494604f3814\": container with ID starting with 9d71793cfc99a6939115dbc087ccc4f2c1f49d5d2013a8facacb0494604f3814 not found: ID does not exist" containerID="9d71793cfc99a6939115dbc087ccc4f2c1f49d5d2013a8facacb0494604f3814"
Jul 29 18:54:51.951013 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.950953    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d71793cfc99a6939115dbc087ccc4f2c1f49d5d2013a8facacb0494604f3814"} err="failed to get container status \"9d71793cfc99a6939115dbc087ccc4f2c1f49d5d2013a8facacb0494604f3814\": rpc error: code = NotFound desc = could not find container \"9d71793cfc99a6939115dbc087ccc4f2c1f49d5d2013a8facacb0494604f3814\": container with ID starting with 9d71793cfc99a6939115dbc087ccc4f2c1f49d5d2013a8facacb0494604f3814 not found: ID does not exist"
Jul 29 18:54:51.955600 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.955577    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"]
Jul 29 18:54:51.959056 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:51.959031    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6df78cd5bb-vptf7"]
Jul 29 18:54:52.687719 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:52.687686    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" path="/var/lib/kubelet/pods/2e8c51da-ebae-465b-8ee6-a047cd78f53f/volumes"
Jul 29 18:54:55.783354 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:55.783319    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-78cc98c464-zhjs8"
Jul 29 18:54:55.830429 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:55.830394    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"]
Jul 29 18:54:55.830689 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:55.830662    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb" podUID="8564439b-a99c-4451-8620-3b37564e8d89" containerName="manager" containerID="cri-o://75b8e22f293f9a7b8103216bb7eb86aede5c36bb7571b86e2efd7770ef37e259" gracePeriod=30
Jul 29 18:54:55.953743 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:55.953710    2560 generic.go:356] "Generic (PLEG): container finished" podID="8564439b-a99c-4451-8620-3b37564e8d89" containerID="75b8e22f293f9a7b8103216bb7eb86aede5c36bb7571b86e2efd7770ef37e259" exitCode=0
Jul 29 18:54:55.953891 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:55.953757    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb" event={"ID":"8564439b-a99c-4451-8620-3b37564e8d89","Type":"ContainerDied","Data":"75b8e22f293f9a7b8103216bb7eb86aede5c36bb7571b86e2efd7770ef37e259"}
Jul 29 18:54:56.108144 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:56.108118    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"
Jul 29 18:54:56.179594 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:56.179561    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9z2s8\" (UniqueName: \"kubernetes.io/projected/8564439b-a99c-4451-8620-3b37564e8d89-kube-api-access-9z2s8\") pod \"8564439b-a99c-4451-8620-3b37564e8d89\" (UID: \"8564439b-a99c-4451-8620-3b37564e8d89\") "
Jul 29 18:54:56.179786 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:56.179634    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8564439b-a99c-4451-8620-3b37564e8d89-cert\") pod \"8564439b-a99c-4451-8620-3b37564e8d89\" (UID: \"8564439b-a99c-4451-8620-3b37564e8d89\") "
Jul 29 18:54:56.181617 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:56.181588    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8564439b-a99c-4451-8620-3b37564e8d89-cert" (OuterVolumeSpecName: "cert") pod "8564439b-a99c-4451-8620-3b37564e8d89" (UID: "8564439b-a99c-4451-8620-3b37564e8d89"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:54:56.181724 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:56.181656    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8564439b-a99c-4451-8620-3b37564e8d89-kube-api-access-9z2s8" (OuterVolumeSpecName: "kube-api-access-9z2s8") pod "8564439b-a99c-4451-8620-3b37564e8d89" (UID: "8564439b-a99c-4451-8620-3b37564e8d89"). InnerVolumeSpecName "kube-api-access-9z2s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:54:56.280562 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:56.280528    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z2s8\" (UniqueName: \"kubernetes.io/projected/8564439b-a99c-4451-8620-3b37564e8d89-kube-api-access-9z2s8\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:56.280562 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:56.280557    2560 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8564439b-a99c-4451-8620-3b37564e8d89-cert\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:54:56.961424 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:56.961342    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"
Jul 29 18:54:56.961850 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:56.961350    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb" event={"ID":"8564439b-a99c-4451-8620-3b37564e8d89","Type":"ContainerDied","Data":"e188400a4d7a65964c017eeadec16f65cd02ba74a1094d5a14e30a649328a099"}
Jul 29 18:54:56.961850 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:56.961474    2560 scope.go:117] "RemoveContainer" containerID="75b8e22f293f9a7b8103216bb7eb86aede5c36bb7571b86e2efd7770ef37e259"
Jul 29 18:54:56.979819 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:56.979789    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"]
Jul 29 18:54:56.983957 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:56.983933    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-6f6f7d67d7-r2nvb"]
Jul 29 18:54:57.884372 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:57.884332    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 29 18:54:58.688604 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:54:58.688567    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8564439b-a99c-4451-8620-3b37564e8d89" path="/var/lib/kubelet/pods/8564439b-a99c-4451-8620-3b37564e8d89/volumes"
Jul 29 18:55:07.884691 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:07.884642    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 29 18:55:17.894978 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:17.894941    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:55:17.903207 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:17.903166    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:55:23.275361 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:23.275321    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"]
Jul 29 18:55:23.275841 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:23.275778    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="main" containerID="cri-o://08c33b9a82f39f5a679dabfe8ba7af7e32dc4132951742187f5b4033fff185d7" gracePeriod=30
Jul 29 18:55:23.321406 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:23.321367    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:55:23.388471 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:23.388410    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:55:24.196056 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.196020    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"]
Jul 29 18:55:24.196548 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.196533    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="main"
Jul 29 18:55:24.196632 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.196551    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="main"
Jul 29 18:55:24.196632 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.196573    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="storage-initializer"
Jul 29 18:55:24.196632 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.196582    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="storage-initializer"
Jul 29 18:55:24.196632 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.196602    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8564439b-a99c-4451-8620-3b37564e8d89" containerName="manager"
Jul 29 18:55:24.196632 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.196611    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8564439b-a99c-4451-8620-3b37564e8d89" containerName="manager"
Jul 29 18:55:24.196899 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.196689    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e8c51da-ebae-465b-8ee6-a047cd78f53f" containerName="main"
Jul 29 18:55:24.196899 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.196704    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8564439b-a99c-4451-8620-3b37564e8d89" containerName="manager"
Jul 29 18:55:24.202150 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.202128    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.204761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.204742    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-77fwd\""
Jul 29 18:55:24.210841 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.210810    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"]
Jul 29 18:55:24.230789 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.230758    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:55:24.326859 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.326823    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-istio-envoy\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.326859 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.326861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-credential-socket\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.327433 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.326881    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-istio-data\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.327433 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.327001    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/489997c7-eab7-4688-9685-91c2d4f18e32-istio-token\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.327433 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.327051    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-workload-certs\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.327433 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.327124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-workload-socket\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.327433 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.327152    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/489997c7-eab7-4688-9685-91c2d4f18e32-istio-podinfo\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.327433 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.327184    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/489997c7-eab7-4688-9685-91c2d4f18e32-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.327433 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.327285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njfsb\" (UniqueName: \"kubernetes.io/projected/489997c7-eab7-4688-9685-91c2d4f18e32-kube-api-access-njfsb\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.428796 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.428753    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-workload-socket\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.428985 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.428813    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/489997c7-eab7-4688-9685-91c2d4f18e32-istio-podinfo\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.428985 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.428847    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/489997c7-eab7-4688-9685-91c2d4f18e32-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.428985 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.428951    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njfsb\" (UniqueName: \"kubernetes.io/projected/489997c7-eab7-4688-9685-91c2d4f18e32-kube-api-access-njfsb\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.429173 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.429002    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-istio-envoy\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.429173 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.429039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-credential-socket\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.429173 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.429063    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-istio-data\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.429173 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.429119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/489997c7-eab7-4688-9685-91c2d4f18e32-istio-token\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.429173 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.429157    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-workload-certs\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.429173 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.429161    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-workload-socket\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.429615 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.429595    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-credential-socket\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.429733 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.429619    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/489997c7-eab7-4688-9685-91c2d4f18e32-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.430222 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.430178    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-workload-certs\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.430376 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.430261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-istio-data\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.431874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.431842    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/489997c7-eab7-4688-9685-91c2d4f18e32-istio-envoy\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.432050 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.432025    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/489997c7-eab7-4688-9685-91c2d4f18e32-istio-podinfo\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.437541 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.437510    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njfsb\" (UniqueName: \"kubernetes.io/projected/489997c7-eab7-4688-9685-91c2d4f18e32-kube-api-access-njfsb\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.438037 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.438010    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/489997c7-eab7-4688-9685-91c2d4f18e32-istio-token\") pod \"router-gateway-2-openshift-default-67f67c9c77-jdf6t\" (UID: \"489997c7-eab7-4688-9685-91c2d4f18e32\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.513797 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.513646    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:24.680927 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.680903    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"]
Jul 29 18:55:24.682903 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:55:24.682870    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489997c7_eab7_4688_9685_91c2d4f18e32.slice/crio-e653a025327d3df1d753c89c3a51264fe8a5e32c622906b9422bb533d007047b WatchSource:0}: Error finding container e653a025327d3df1d753c89c3a51264fe8a5e32c622906b9422bb533d007047b: Status 404 returned error can't find the container with id e653a025327d3df1d753c89c3a51264fe8a5e32c622906b9422bb533d007047b
Jul 29 18:55:24.685018 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.684976    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:55:24.685137 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.685067    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:55:24.685137 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:24.685110    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:55:25.124422 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:25.124379    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t" event={"ID":"489997c7-eab7-4688-9685-91c2d4f18e32","Type":"ContainerStarted","Data":"d83d72a10e0e0246b74f318b903fa1ea6eca262b4b8c04f6d240bd4e5eda642f"}
Jul 29 18:55:25.124641 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:25.124449    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t" event={"ID":"489997c7-eab7-4688-9685-91c2d4f18e32","Type":"ContainerStarted","Data":"e653a025327d3df1d753c89c3a51264fe8a5e32c622906b9422bb533d007047b"}
Jul 29 18:55:25.145868 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:25.145814    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t" podStartSLOduration=1.145797638 podStartE2EDuration="1.145797638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:55:25.144821803 +0000 UTC m=+1251.038822223" watchObservedRunningTime="2026-07-29 18:55:25.145797638 +0000 UTC m=+1251.039798037"
Jul 29 18:55:25.515781 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:25.515331    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:25.519799 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:25.519777    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:25.959451 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:25.959328    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:55:26.130097 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:26.130067    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:26.130951 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:26.130929    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:27.371305 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:27.371269    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:55:35.183804 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:35.183721    2560 generic.go:356] "Generic (PLEG): container finished" podID="489997c7-eab7-4688-9685-91c2d4f18e32" containerID="d83d72a10e0e0246b74f318b903fa1ea6eca262b4b8c04f6d240bd4e5eda642f" exitCode=0
Jul 29 18:55:35.184329 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:35.183803    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t" event={"ID":"489997c7-eab7-4688-9685-91c2d4f18e32","Type":"ContainerDied","Data":"d83d72a10e0e0246b74f318b903fa1ea6eca262b4b8c04f6d240bd4e5eda642f"}
Jul 29 18:55:35.184329 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:35.184311    2560 scope.go:117] "RemoveContainer" containerID="d83d72a10e0e0246b74f318b903fa1ea6eca262b4b8c04f6d240bd4e5eda642f"
Jul 29 18:55:35.185335 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:35.185309    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:55:35.185451 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:35.185381    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:55:35.185451 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:35.185424    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 18:55:36.190599 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:36.190550    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t" event={"ID":"489997c7-eab7-4688-9685-91c2d4f18e32","Type":"ContainerStarted","Data":"db3bae859df1434928a0ba4efb1de40fc58e6ad05abed1088a24ea6a03c97fed"}
Jul 29 18:55:36.515343 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:36.515246    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:36.515526 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:36.515330    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t" podUID="489997c7-eab7-4688-9685-91c2d4f18e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 29 18:55:37.514159 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:37.514114    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t" podUID="489997c7-eab7-4688-9685-91c2d4f18e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 29 18:55:38.514490 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:38.514436    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t" podUID="489997c7-eab7-4688-9685-91c2d4f18e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 29 18:55:39.279170 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.279131    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"]
Jul 29 18:55:39.282485 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.282464    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.285377 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.285355    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 29 18:55:39.321573 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.321525    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"]
Jul 29 18:55:39.323594 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.323562    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:55:39.360423 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.360390    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-home\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.360423 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.360426    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-model-cache\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.360674 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.360501    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-dshm\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.360674 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.360534    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-tmp-dir\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.360674 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.360567    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade8b322-0eea-4969-8866-71c0dd7b265e-tls-certs\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.360674 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.360592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-kserve-provision-location\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.360863 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.360738    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcxkf\" (UniqueName: \"kubernetes.io/projected/ade8b322-0eea-4969-8866-71c0dd7b265e-kube-api-access-rcxkf\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.461147 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.461106    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-dshm\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.461147 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.461149    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-tmp-dir\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.461417 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.461167    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade8b322-0eea-4969-8866-71c0dd7b265e-tls-certs\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.461417 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.461187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-kserve-provision-location\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.461417 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.461258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcxkf\" (UniqueName: \"kubernetes.io/projected/ade8b322-0eea-4969-8866-71c0dd7b265e-kube-api-access-rcxkf\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.461588 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.461461    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-home\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.461588 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.461507    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-model-cache\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.461713 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.461682    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-tmp-dir\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.461773 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.461733    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-kserve-provision-location\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.461842 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.461814    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-home\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.461923 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.461904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-model-cache\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.463466 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.463445    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-dshm\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.463860 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.463837    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade8b322-0eea-4969-8866-71c0dd7b265e-tls-certs\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.469181 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.469160    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcxkf\" (UniqueName: \"kubernetes.io/projected/ade8b322-0eea-4969-8866-71c0dd7b265e-kube-api-access-rcxkf\") pod \"router-with-refs-test-kserve-7c5b94f5c4-kjmqk\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.515246 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.515122    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t" podUID="489997c7-eab7-4688-9685-91c2d4f18e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 29 18:55:39.607148 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.606835    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:39.675664 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.675576    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:55:39.779372 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:39.779341    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"]
Jul 29 18:55:39.780560 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:55:39.780533    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade8b322_0eea_4969_8866_71c0dd7b265e.slice/crio-7c7d74d20ba27d7c98c0b038a6eedd27421209799b4ff6cf7ea0459418a1d03c WatchSource:0}: Error finding container 7c7d74d20ba27d7c98c0b038a6eedd27421209799b4ff6cf7ea0459418a1d03c: Status 404 returned error can't find the container with id 7c7d74d20ba27d7c98c0b038a6eedd27421209799b4ff6cf7ea0459418a1d03c
Jul 29 18:55:40.227964 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:40.227924    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" event={"ID":"ade8b322-0eea-4969-8866-71c0dd7b265e","Type":"ContainerStarted","Data":"a90fbe4f89d3f0acdbb18cda2c574e25cf9aa26484429ee8c1994743fa88f928"}
Jul 29 18:55:40.228131 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:40.227996    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" event={"ID":"ade8b322-0eea-4969-8866-71c0dd7b265e","Type":"ContainerStarted","Data":"7c7d74d20ba27d7c98c0b038a6eedd27421209799b4ff6cf7ea0459418a1d03c"}
Jul 29 18:55:40.515209 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:40.515099    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t" podUID="489997c7-eab7-4688-9685-91c2d4f18e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 29 18:55:41.130851 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:41.130813    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:41.518408 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:41.518329    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:41.519842 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:41.519817    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-67f67c9c77-jdf6t"
Jul 29 18:55:44.900995 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:44.900960    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"]
Jul 29 18:55:44.904881 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:44.904856    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:44.907820 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:44.907796    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 29 18:55:44.907952 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:44.907768    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-gfvmv\""
Jul 29 18:55:44.919238 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:44.919121    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"]
Jul 29 18:55:44.937440 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:44.937407    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"]
Jul 29 18:55:44.940002 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:44.939974    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:44.955603 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:44.955578    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"]
Jul 29 18:55:44.963570 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:44.963545    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:55:45.008091 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.008056    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npnr4\" (UniqueName: \"kubernetes.io/projected/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-kube-api-access-npnr4\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.008277 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.008102    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-dshm\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.008277 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.008127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-home\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.008277 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.008151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.008277 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.008190    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.008277 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.008236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.008457 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.008277    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.008457 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.008310    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.008457 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.008342    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.008457 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.008446    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-model-cache\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.008581 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.008485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thn77\" (UniqueName: \"kubernetes.io/projected/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-kube-api-access-thn77\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.008581 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.008540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.008642 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.008578    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-tls-certs\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.008642 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.008604    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-home\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.109696 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.109662    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thn77\" (UniqueName: \"kubernetes.io/projected/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-kube-api-access-thn77\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.109893 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.109709    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.109893 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.109729    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-tls-certs\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.109893 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.109747    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-home\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.109893 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.109842    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npnr4\" (UniqueName: \"kubernetes.io/projected/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-kube-api-access-npnr4\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.110122 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.109907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-dshm\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.110122 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.109933    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-home\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.110122 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.109960    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.110122 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.110017    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.110122 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.110052    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.110425 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.110132    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-home\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.110425 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.110179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.110425 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.110181    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.110425 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.110226    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.110425 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.110270    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.110425 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.110342    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-home\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.110425 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.110349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-model-cache\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.110801 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.110430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.110801 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.110491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.110801 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.110577    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-model-cache\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.110957 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.110850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.110957 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.110857    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.112623 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.112592    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.112755 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.112727    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-dshm\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.112933 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.112912    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-tls-certs\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.113064 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.113044    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.117612 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.117588    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npnr4\" (UniqueName: \"kubernetes.io/projected/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-kube-api-access-npnr4\") pod \"router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.117844 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.117824    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thn77\" (UniqueName: \"kubernetes.io/projected/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-kube-api-access-thn77\") pod \"router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.218136 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.218044    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:45.250915 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.250879    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:45.376717 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.376689    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"]
Jul 29 18:55:45.379110 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:55:45.379080    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd430a332_a0fb_4a6a_bce7_9cc0d1b21208.slice/crio-0dc3ea0599521cb6edf0db8d70978453b515c06004586aeb58c5cfaa1d49b9bb WatchSource:0}: Error finding container 0dc3ea0599521cb6edf0db8d70978453b515c06004586aeb58c5cfaa1d49b9bb: Status 404 returned error can't find the container with id 0dc3ea0599521cb6edf0db8d70978453b515c06004586aeb58c5cfaa1d49b9bb
Jul 29 18:55:45.419880 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.419859    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"]
Jul 29 18:55:45.421556 ip-10-0-137-52 kubenswrapper[2560]: W0729 18:55:45.421530    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff6225f_5f30_4745_a2d8_65d4ef07a5f9.slice/crio-614cb709c42fb5a7ab7ebf01993a6d82d6704aebc05ef8c953aff2de7a055dc0 WatchSource:0}: Error finding container 614cb709c42fb5a7ab7ebf01993a6d82d6704aebc05ef8c953aff2de7a055dc0: Status 404 returned error can't find the container with id 614cb709c42fb5a7ab7ebf01993a6d82d6704aebc05ef8c953aff2de7a055dc0
Jul 29 18:55:45.538279 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:45.538248    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:55:46.266635 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:46.266599    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" event={"ID":"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9","Type":"ContainerStarted","Data":"cb7941eb390bbe892d184f3306c26907556ae59ed5541a60bb94e0233a68a831"}
Jul 29 18:55:46.267104 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:46.266660    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" event={"ID":"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9","Type":"ContainerStarted","Data":"614cb709c42fb5a7ab7ebf01993a6d82d6704aebc05ef8c953aff2de7a055dc0"}
Jul 29 18:55:46.268232 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:46.268184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" event={"ID":"d430a332-a0fb-4a6a-bce7-9cc0d1b21208","Type":"ContainerStarted","Data":"dd20029937fc6149a9dba77d72f9d6fe3b53c23e49d071cc4036d6358f125e4d"}
Jul 29 18:55:46.268513 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:46.268250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" event={"ID":"d430a332-a0fb-4a6a-bce7-9cc0d1b21208","Type":"ContainerStarted","Data":"0dc3ea0599521cb6edf0db8d70978453b515c06004586aeb58c5cfaa1d49b9bb"}
Jul 29 18:55:46.268513 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:46.268294    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:46.269783 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:46.269761    2560 generic.go:356] "Generic (PLEG): container finished" podID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerID="a90fbe4f89d3f0acdbb18cda2c574e25cf9aa26484429ee8c1994743fa88f928" exitCode=0
Jul 29 18:55:46.269888 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:46.269823    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" event={"ID":"ade8b322-0eea-4969-8866-71c0dd7b265e","Type":"ContainerDied","Data":"a90fbe4f89d3f0acdbb18cda2c574e25cf9aa26484429ee8c1994743fa88f928"}
Jul 29 18:55:47.278002 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:47.277959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" event={"ID":"d430a332-a0fb-4a6a-bce7-9cc0d1b21208","Type":"ContainerStarted","Data":"c41909a05ecd4371fdd0fb41417eeea6bbdeda27af5200062c083a94465301a7"}
Jul 29 18:55:47.279854 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:47.279813    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" event={"ID":"ade8b322-0eea-4969-8866-71c0dd7b265e","Type":"ContainerStarted","Data":"2fdb0e9a9d37e9cf10b5c5be96b2ba5ec7c52098f1884fef22016050028cdca1"}
Jul 29 18:55:47.327863 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:47.327804    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podStartSLOduration=2.6965616580000003 podStartE2EDuration="8.327787957s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.631226299s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:55:47.32528516 +0000 UTC m=+1273.219285581" watchObservedRunningTime="2026-07-29 18:55:47.327787957 +0000 UTC m=+1273.221788358"
Jul 29 18:55:49.607523 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:49.607480    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:49.608010 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:49.607607    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:55:49.609337 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:49.609299    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:55:52.319361 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:52.319321    2560 generic.go:356] "Generic (PLEG): container finished" podID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerID="cb7941eb390bbe892d184f3306c26907556ae59ed5541a60bb94e0233a68a831" exitCode=0
Jul 29 18:55:52.319782 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:52.319397    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" event={"ID":"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9","Type":"ContainerDied","Data":"cb7941eb390bbe892d184f3306c26907556ae59ed5541a60bb94e0233a68a831"}
Jul 29 18:55:52.321505 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:52.321481    2560 generic.go:356] "Generic (PLEG): container finished" podID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerID="c41909a05ecd4371fdd0fb41417eeea6bbdeda27af5200062c083a94465301a7" exitCode=0
Jul 29 18:55:52.321616 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:52.321554    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" event={"ID":"d430a332-a0fb-4a6a-bce7-9cc0d1b21208","Type":"ContainerDied","Data":"c41909a05ecd4371fdd0fb41417eeea6bbdeda27af5200062c083a94465301a7"}
Jul 29 18:55:53.330861 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.330829    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" event={"ID":"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9","Type":"ContainerStarted","Data":"79dc79d490ce5604df97f11fc602d40fbb79f66ae3f76626007c48c85618e7d3"}
Jul 29 18:55:53.333986 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.333955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" event={"ID":"d430a332-a0fb-4a6a-bce7-9cc0d1b21208","Type":"ContainerStarted","Data":"b40c8efe347eb3f3054b1e26caaf49e63282aa38bcac20ba10e7172a2eeba953"}
Jul 29 18:55:53.355467 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.355397    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podStartSLOduration=3.485807593 podStartE2EDuration="9.355376294s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.869568701s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:55:53.352731422 +0000 UTC m=+1279.246731850" watchObservedRunningTime="2026-07-29 18:55:53.355376294 +0000 UTC m=+1279.249376696"
Jul 29 18:55:53.377301 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.377051    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podStartSLOduration=3.105554902 podStartE2EDuration="9.377028642s" totalImagesPullingTime="764.861625ms" totalInitContainerRuntime="5.506612115s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:55:53.373553992 +0000 UTC m=+1279.267554400" watchObservedRunningTime="2026-07-29 18:55:53.377028642 +0000 UTC m=+1279.271029046"
Jul 29 18:55:53.590617 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.590587    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-c7b9f676b-r2k65_128383ad-4b37-4851-adfa-149221d186cc/main/0.log"
Jul 29 18:55:53.613172 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.613137    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:55:53.702877 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.702837    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-kserve-provision-location\") pod \"128383ad-4b37-4851-adfa-149221d186cc\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") "
Jul 29 18:55:53.702877 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.702879    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-model-cache\") pod \"128383ad-4b37-4851-adfa-149221d186cc\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") "
Jul 29 18:55:53.703149 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.702918    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-tmp-dir\") pod \"128383ad-4b37-4851-adfa-149221d186cc\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") "
Jul 29 18:55:53.703149 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.702980    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/128383ad-4b37-4851-adfa-149221d186cc-tls-certs\") pod \"128383ad-4b37-4851-adfa-149221d186cc\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") "
Jul 29 18:55:53.703149 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.703032    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-home\") pod \"128383ad-4b37-4851-adfa-149221d186cc\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") "
Jul 29 18:55:53.703149 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.703090    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5vfq\" (UniqueName: \"kubernetes.io/projected/128383ad-4b37-4851-adfa-149221d186cc-kube-api-access-n5vfq\") pod \"128383ad-4b37-4851-adfa-149221d186cc\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") "
Jul 29 18:55:53.703149 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.703117    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-dshm\") pod \"128383ad-4b37-4851-adfa-149221d186cc\" (UID: \"128383ad-4b37-4851-adfa-149221d186cc\") "
Jul 29 18:55:53.703451 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.703208    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "128383ad-4b37-4851-adfa-149221d186cc" (UID: "128383ad-4b37-4851-adfa-149221d186cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:55:53.703451 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.703431    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:55:53.706061 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.704011    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-home" (OuterVolumeSpecName: "home") pod "128383ad-4b37-4851-adfa-149221d186cc" (UID: "128383ad-4b37-4851-adfa-149221d186cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:55:53.706061 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.705862    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-dshm" (OuterVolumeSpecName: "dshm") pod "128383ad-4b37-4851-adfa-149221d186cc" (UID: "128383ad-4b37-4851-adfa-149221d186cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:55:53.706061 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.706009    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/128383ad-4b37-4851-adfa-149221d186cc-kube-api-access-n5vfq" (OuterVolumeSpecName: "kube-api-access-n5vfq") pod "128383ad-4b37-4851-adfa-149221d186cc" (UID: "128383ad-4b37-4851-adfa-149221d186cc"). InnerVolumeSpecName "kube-api-access-n5vfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:55:53.706617 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.706581    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/128383ad-4b37-4851-adfa-149221d186cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "128383ad-4b37-4851-adfa-149221d186cc" (UID: "128383ad-4b37-4851-adfa-149221d186cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:55:53.720587 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.720548    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "128383ad-4b37-4851-adfa-149221d186cc" (UID: "128383ad-4b37-4851-adfa-149221d186cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:55:53.752266 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.752182    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "128383ad-4b37-4851-adfa-149221d186cc" (UID: "128383ad-4b37-4851-adfa-149221d186cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:55:53.804875 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.804840    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5vfq\" (UniqueName: \"kubernetes.io/projected/128383ad-4b37-4851-adfa-149221d186cc-kube-api-access-n5vfq\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:55:53.804875 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.804870    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:55:53.804875 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.804880    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:55:53.805189 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.804889    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:55:53.805189 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.804902    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/128383ad-4b37-4851-adfa-149221d186cc-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:55:53.805189 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:53.804913    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/128383ad-4b37-4851-adfa-149221d186cc-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 18:55:54.343583 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:54.343553    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-c7b9f676b-r2k65_128383ad-4b37-4851-adfa-149221d186cc/main/0.log"
Jul 29 18:55:54.344147 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:54.343971    2560 generic.go:356] "Generic (PLEG): container finished" podID="128383ad-4b37-4851-adfa-149221d186cc" containerID="08c33b9a82f39f5a679dabfe8ba7af7e32dc4132951742187f5b4033fff185d7" exitCode=137
Jul 29 18:55:54.344147 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:54.344029    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"
Jul 29 18:55:54.344147 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:54.344065    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" event={"ID":"128383ad-4b37-4851-adfa-149221d186cc","Type":"ContainerDied","Data":"08c33b9a82f39f5a679dabfe8ba7af7e32dc4132951742187f5b4033fff185d7"}
Jul 29 18:55:54.344147 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:54.344143    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65" event={"ID":"128383ad-4b37-4851-adfa-149221d186cc","Type":"ContainerDied","Data":"bf5e6b14b889530b6a81644c9f6f39dba82d6ee88ebb65e5823a5b835daac190"}
Jul 29 18:55:54.344409 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:54.344176    2560 scope.go:117] "RemoveContainer" containerID="08c33b9a82f39f5a679dabfe8ba7af7e32dc4132951742187f5b4033fff185d7"
Jul 29 18:55:54.357332 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:54.357308    2560 scope.go:117] "RemoveContainer" containerID="4c5c86328969e40d8a36a5ca5cf08aea0cd04ec48614b1d93ed1aa3df8f438eb"
Jul 29 18:55:54.371402 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:54.371124    2560 scope.go:117] "RemoveContainer" containerID="08c33b9a82f39f5a679dabfe8ba7af7e32dc4132951742187f5b4033fff185d7"
Jul 29 18:55:54.371651 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:55:54.371621    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c33b9a82f39f5a679dabfe8ba7af7e32dc4132951742187f5b4033fff185d7\": container with ID starting with 08c33b9a82f39f5a679dabfe8ba7af7e32dc4132951742187f5b4033fff185d7 not found: ID does not exist" containerID="08c33b9a82f39f5a679dabfe8ba7af7e32dc4132951742187f5b4033fff185d7"
Jul 29 18:55:54.371761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:54.371660    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c33b9a82f39f5a679dabfe8ba7af7e32dc4132951742187f5b4033fff185d7"} err="failed to get container status \"08c33b9a82f39f5a679dabfe8ba7af7e32dc4132951742187f5b4033fff185d7\": rpc error: code = NotFound desc = could not find container \"08c33b9a82f39f5a679dabfe8ba7af7e32dc4132951742187f5b4033fff185d7\": container with ID starting with 08c33b9a82f39f5a679dabfe8ba7af7e32dc4132951742187f5b4033fff185d7 not found: ID does not exist"
Jul 29 18:55:54.371761 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:54.371679    2560 scope.go:117] "RemoveContainer" containerID="4c5c86328969e40d8a36a5ca5cf08aea0cd04ec48614b1d93ed1aa3df8f438eb"
Jul 29 18:55:54.372009 ip-10-0-137-52 kubenswrapper[2560]: E0729 18:55:54.371976    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c5c86328969e40d8a36a5ca5cf08aea0cd04ec48614b1d93ed1aa3df8f438eb\": container with ID starting with 4c5c86328969e40d8a36a5ca5cf08aea0cd04ec48614b1d93ed1aa3df8f438eb not found: ID does not exist" containerID="4c5c86328969e40d8a36a5ca5cf08aea0cd04ec48614b1d93ed1aa3df8f438eb"
Jul 29 18:55:54.372135 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:54.372018    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c5c86328969e40d8a36a5ca5cf08aea0cd04ec48614b1d93ed1aa3df8f438eb"} err="failed to get container status \"4c5c86328969e40d8a36a5ca5cf08aea0cd04ec48614b1d93ed1aa3df8f438eb\": rpc error: code = NotFound desc = could not find container \"4c5c86328969e40d8a36a5ca5cf08aea0cd04ec48614b1d93ed1aa3df8f438eb\": container with ID starting with 4c5c86328969e40d8a36a5ca5cf08aea0cd04ec48614b1d93ed1aa3df8f438eb not found: ID does not exist"
Jul 29 18:55:54.374052 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:54.374026    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"]
Jul 29 18:55:54.378892 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:54.378864    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-c7b9f676b-r2k65"]
Jul 29 18:55:54.688689 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:54.688597    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="128383ad-4b37-4851-adfa-149221d186cc" path="/var/lib/kubelet/pods/128383ad-4b37-4851-adfa-149221d186cc/volumes"
Jul 29 18:55:55.218224 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:55.218158    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:55.218518 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:55.218251    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:55:55.220157 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:55.220126    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:55:55.251406 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:55.251366    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:55.251600 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:55.251482    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:55:55.253240 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:55.253176    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:55:59.608612 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:59.608563    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:55:59.769529 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:55:59.769485    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:56:05.219681 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:05.218720    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:56:05.249709 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:05.249673    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:56:05.251476 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:05.251441    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:56:09.608870 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:09.608817    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:56:15.218655 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:15.218594    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:56:15.252279 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:15.252225    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:56:19.608040 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:19.607994    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:56:25.218877 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:25.218823    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:56:25.252257 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:25.252178    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:56:29.608190 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:29.608140    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:56:30.435867 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:30.435823    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:56:30.539979 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:30.539935    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:56:35.219455 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:35.219408    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:56:35.251890 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:35.251843    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:56:39.607743 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:39.607688    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:56:45.218874 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:45.218823    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:56:45.251682 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:45.251632    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:56:49.608602 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:49.608554    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:56:55.219519 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:55.219460    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:56:55.251699 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:55.251660    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:56:59.608423 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:56:59.608320    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:57:05.218950 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:05.218894    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:57:05.251788 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:05.251739    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:57:09.608383 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:09.608338    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:57:15.219214 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:15.219150    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:57:15.252266 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:15.252228    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:57:19.608343 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:19.608290    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:57:23.465479 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:23.465436    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:57:23.567245 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:23.567192    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:57:25.218471 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:25.218427    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:57:25.252172 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:25.252133    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:57:29.607891 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:29.607848    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:57:35.219379 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:35.219329    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:57:35.251329 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:35.251292    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:57:39.608446 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:39.608400    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:57:45.219161 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:45.219107    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:57:45.252208 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:45.252161    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:57:49.608291 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:49.608247    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:57:55.219165 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:55.219112    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:57:55.252143 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:55.252101    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:57:59.608669 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:57:59.608628    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:58:05.219474 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:05.219417    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:58:05.252011 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:05.251966    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:58:09.608167 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:09.608124    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:58:15.219225 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:15.219147    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:58:15.251262 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:15.251223    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:58:19.608139 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:19.608091    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:58:25.218479 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:25.218429    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:58:25.251683 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:25.251637    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:58:29.607999 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:29.607902    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 29 18:58:35.219278 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:35.219212    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:58:35.251361 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:35.251316    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:58:39.617877 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:39.617843    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:58:39.625733 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:39.625699    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 18:58:45.219220 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:45.219145    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 29 18:58:45.251832 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:45.251784    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 18:58:55.231354 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:55.231318    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:58:55.251190 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:55.251157    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 18:58:55.262145 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:55.262114    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:58:55.269626 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:58:55.269602    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 18:59:30.751117 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:59:30.751083    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 18:59:34.612334 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:59:34.612300    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 18:59:34.615059 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:59:34.615038    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 18:59:43.852657 ip-10-0-137-52 kubenswrapper[2560]: I0729 18:59:43.852624    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:01:29.249260 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:01:29.249149    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:04:34.649609 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:04:34.649579    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 19:04:34.653721 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:04:34.653697    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 19:05:35.052928 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:05:35.052890    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:06:08.699052 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:06:08.698963    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:08:21.506719 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:08:21.506680    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:09:34.688170 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:09:34.688137    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 19:09:34.695955 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:09:34.695930    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 19:10:09.115874 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:09.115840    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:10:26.406872 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:26.406785    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"]
Jul 29 19:10:26.407305 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:26.407021    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main" containerID="cri-o://2fdb0e9a9d37e9cf10b5c5be96b2ba5ec7c52098f1884fef22016050028cdca1" gracePeriod=30
Jul 29 19:10:26.415432 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:26.415407    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:10:26.500190 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:26.500157    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:10:30.502911 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:30.502873    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:10:45.129034 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.128986    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"]
Jul 29 19:10:45.129779 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.129440    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="main"
Jul 29 19:10:45.129779 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.129456    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="main"
Jul 29 19:10:45.129779 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.129485    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="storage-initializer"
Jul 29 19:10:45.129779 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.129492    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="storage-initializer"
Jul 29 19:10:45.129779 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.129549    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="128383ad-4b37-4851-adfa-149221d186cc" containerName="main"
Jul 29 19:10:45.132675 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.132652    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"]
Jul 29 19:10:45.132796 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.132775    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.136079 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.136061    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.140767 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.140753    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-qr7g6\""
Jul 29 19:10:45.141752 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.141737    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
Jul 29 19:10:45.168346 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.168320    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tq4v\" (UniqueName: \"kubernetes.io/projected/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-kube-api-access-5tq4v\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.168496 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.168351    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.168496 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.168375    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.168496 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.168394    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.168496 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.168474    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.168720 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.168495    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.168720 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.168521    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff86q\" (UniqueName: \"kubernetes.io/projected/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-kube-api-access-ff86q\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.168720 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.168573    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.168720 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.168626    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.168720 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.168667    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.168720 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.168715    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.168959 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.168785    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.168959 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.168802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.168959 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.168818    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.187795 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.187767    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"]
Jul 29 19:10:45.225345 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.225316    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"]
Jul 29 19:10:45.231600 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.231570    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:10:45.269834 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.269802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.270008 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.269838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.270008 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.269858    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.270008 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.269878    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tq4v\" (UniqueName: \"kubernetes.io/projected/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-kube-api-access-5tq4v\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.270008 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.269894    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.270008 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.269926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.270008 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.269945    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.270008 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.269981    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.270457 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.270233    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.270457 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.270275    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff86q\" (UniqueName: \"kubernetes.io/projected/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-kube-api-access-ff86q\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.270457 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.270311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.270457 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.270349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.270457 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.270395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.270457 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.270401    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.270457 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.270425    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.270457 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.270444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.270881 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.270468    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.270881 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.270743    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.270881 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.270810    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.270881 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.270819    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.271097 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.270949    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.271097 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.270995    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.272513 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.272490    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.272797 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.272778    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.272843 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.272801    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.273086 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.273071    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.292094 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.292072    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tq4v\" (UniqueName: \"kubernetes.io/projected/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-kube-api-access-5tq4v\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.311029 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.311004    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff86q\" (UniqueName: \"kubernetes.io/projected/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-kube-api-access-ff86q\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.443634 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.443543    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:45.449840 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.449810    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:45.659539 ip-10-0-137-52 kubenswrapper[2560]: W0729 19:10:45.659508    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e25e2f1_d4d0_47c5_a8e8_67c0c4a7de0d.slice/crio-0a1e7af5797177f640160d6b3e7b306cf6bf3556c44919e72026e9f88a9165a5 WatchSource:0}: Error finding container 0a1e7af5797177f640160d6b3e7b306cf6bf3556c44919e72026e9f88a9165a5: Status 404 returned error can't find the container with id 0a1e7af5797177f640160d6b3e7b306cf6bf3556c44919e72026e9f88a9165a5
Jul 29 19:10:45.668289 ip-10-0-137-52 kubenswrapper[2560]: W0729 19:10:45.668265    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d93eef3_4de0_4a8d_ada0_cc477c5c50b2.slice/crio-021e1175404a3d7981fbce305f6898c4d8c68022775f30f8564ef3cb4b7e5452 WatchSource:0}: Error finding container 021e1175404a3d7981fbce305f6898c4d8c68022775f30f8564ef3cb4b7e5452: Status 404 returned error can't find the container with id 021e1175404a3d7981fbce305f6898c4d8c68022775f30f8564ef3cb4b7e5452
Jul 29 19:10:45.671428 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.671407    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"]
Jul 29 19:10:45.673951 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:45.673928    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"]
Jul 29 19:10:46.388426 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:46.388374    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" event={"ID":"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d","Type":"ContainerStarted","Data":"067a415ecf7952aa79a7c41b133b9c171c34a4ec0fe5637c697e0c94fe56de52"}
Jul 29 19:10:46.388920 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:46.388438    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" event={"ID":"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d","Type":"ContainerStarted","Data":"0a1e7af5797177f640160d6b3e7b306cf6bf3556c44919e72026e9f88a9165a5"}
Jul 29 19:10:46.390814 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:46.390778    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" event={"ID":"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2","Type":"ContainerStarted","Data":"e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de"}
Jul 29 19:10:46.390954 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:46.390836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" event={"ID":"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2","Type":"ContainerStarted","Data":"021e1175404a3d7981fbce305f6898c4d8c68022775f30f8564ef3cb4b7e5452"}
Jul 29 19:10:46.391581 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:46.391559    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:46.619220 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:46.616752    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:10:47.410776 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:47.410732    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" event={"ID":"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2","Type":"ContainerStarted","Data":"1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29"}
Jul 29 19:10:48.148913 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:48.148864    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"]
Jul 29 19:10:48.149169 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:48.149116    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main" containerID="cri-o://79dc79d490ce5604df97f11fc602d40fbb79f66ae3f76626007c48c85618e7d3" gracePeriod=30
Jul 29 19:10:48.153551 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:48.153517    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:10:48.158785 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:48.158719    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"]
Jul 29 19:10:48.159361 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:48.159325    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main" containerID="cri-o://b40c8efe347eb3f3054b1e26caaf49e63282aa38bcac20ba10e7172a2eeba953" gracePeriod=30
Jul 29 19:10:48.344821 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:48.344778    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:10:52.447438 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:52.447397    2560 generic.go:356] "Generic (PLEG): container finished" podID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerID="067a415ecf7952aa79a7c41b133b9c171c34a4ec0fe5637c697e0c94fe56de52" exitCode=0
Jul 29 19:10:52.447996 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:52.447471    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" event={"ID":"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d","Type":"ContainerDied","Data":"067a415ecf7952aa79a7c41b133b9c171c34a4ec0fe5637c697e0c94fe56de52"}
Jul 29 19:10:53.455618 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:53.455517    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" event={"ID":"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d","Type":"ContainerStarted","Data":"2240c0d54af8fe9ba73cd247c6548708b3d233ddba34aaf00dd526aa402b2ed4"}
Jul 29 19:10:53.457383 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:53.457356    2560 generic.go:356] "Generic (PLEG): container finished" podID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerID="1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29" exitCode=0
Jul 29 19:10:53.457501 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:53.457435    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" event={"ID":"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2","Type":"ContainerDied","Data":"1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29"}
Jul 29 19:10:53.492846 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:53.492789    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podStartSLOduration=3.449086697 podStartE2EDuration="9.492769449s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.043682752s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:10:53.490828057 +0000 UTC m=+2179.384828479" watchObservedRunningTime="2026-07-29 19:10:53.492769449 +0000 UTC m=+2179.386769855"
Jul 29 19:10:54.467982 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:54.467939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" event={"ID":"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2","Type":"ContainerStarted","Data":"0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1"}
Jul 29 19:10:54.502919 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:54.502851    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podStartSLOduration=3.741231074 podStartE2EDuration="10.502828499s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.761597425s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:10:54.496626215 +0000 UTC m=+2180.390626649" watchObservedRunningTime="2026-07-29 19:10:54.502828499 +0000 UTC m=+2180.396828901"
Jul 29 19:10:55.443837 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:55.443777    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:55.444100 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:55.443882    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:55.445447 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:55.445409    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:10:55.450062 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:55.450033    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:55.450192 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:55.450100    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:10:55.451219 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:55.451172    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:10:55.486084 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:55.486060    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:10:56.830320 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.830287    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-7c5b94f5c4-kjmqk_ade8b322-0eea-4969-8866-71c0dd7b265e/main/0.log"
Jul 29 19:10:56.855263 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.855228    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 19:10:56.899629 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.899595    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-model-cache\") pod \"ade8b322-0eea-4969-8866-71c0dd7b265e\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") "
Jul 29 19:10:56.899841 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.899657    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-home\") pod \"ade8b322-0eea-4969-8866-71c0dd7b265e\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") "
Jul 29 19:10:56.899841 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.899721    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-tmp-dir\") pod \"ade8b322-0eea-4969-8866-71c0dd7b265e\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") "
Jul 29 19:10:56.899841 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.899770    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade8b322-0eea-4969-8866-71c0dd7b265e-tls-certs\") pod \"ade8b322-0eea-4969-8866-71c0dd7b265e\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") "
Jul 29 19:10:56.899841 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.899808    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-dshm\") pod \"ade8b322-0eea-4969-8866-71c0dd7b265e\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") "
Jul 29 19:10:56.900074 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.899851    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcxkf\" (UniqueName: \"kubernetes.io/projected/ade8b322-0eea-4969-8866-71c0dd7b265e-kube-api-access-rcxkf\") pod \"ade8b322-0eea-4969-8866-71c0dd7b265e\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") "
Jul 29 19:10:56.900074 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.899861    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ade8b322-0eea-4969-8866-71c0dd7b265e" (UID: "ade8b322-0eea-4969-8866-71c0dd7b265e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:10:56.900074 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.899887    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-kserve-provision-location\") pod \"ade8b322-0eea-4969-8866-71c0dd7b265e\" (UID: \"ade8b322-0eea-4969-8866-71c0dd7b265e\") "
Jul 29 19:10:56.900314 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.900286    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:10:56.901499 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.901467    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-home" (OuterVolumeSpecName: "home") pod "ade8b322-0eea-4969-8866-71c0dd7b265e" (UID: "ade8b322-0eea-4969-8866-71c0dd7b265e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:10:56.921876 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.921836    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ade8b322-0eea-4969-8866-71c0dd7b265e" (UID: "ade8b322-0eea-4969-8866-71c0dd7b265e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:10:56.924885 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.924861    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-dshm" (OuterVolumeSpecName: "dshm") pod "ade8b322-0eea-4969-8866-71c0dd7b265e" (UID: "ade8b322-0eea-4969-8866-71c0dd7b265e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:10:56.931365 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.931332    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade8b322-0eea-4969-8866-71c0dd7b265e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ade8b322-0eea-4969-8866-71c0dd7b265e" (UID: "ade8b322-0eea-4969-8866-71c0dd7b265e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:10:56.931651 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.931574    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ade8b322-0eea-4969-8866-71c0dd7b265e" (UID: "ade8b322-0eea-4969-8866-71c0dd7b265e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:10:56.931766 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:56.931739    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade8b322-0eea-4969-8866-71c0dd7b265e-kube-api-access-rcxkf" (OuterVolumeSpecName: "kube-api-access-rcxkf") pod "ade8b322-0eea-4969-8866-71c0dd7b265e" (UID: "ade8b322-0eea-4969-8866-71c0dd7b265e"). InnerVolumeSpecName "kube-api-access-rcxkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:10:57.001212 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.001105    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:10:57.001212 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.001149    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:10:57.001212 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.001164    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade8b322-0eea-4969-8866-71c0dd7b265e-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:10:57.001212 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.001177    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:10:57.001212 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.001192    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcxkf\" (UniqueName: \"kubernetes.io/projected/ade8b322-0eea-4969-8866-71c0dd7b265e-kube-api-access-rcxkf\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:10:57.001494 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.001224    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade8b322-0eea-4969-8866-71c0dd7b265e-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:10:57.485979 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.485945    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-7c5b94f5c4-kjmqk_ade8b322-0eea-4969-8866-71c0dd7b265e/main/0.log"
Jul 29 19:10:57.486415 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.486388    2560 generic.go:356] "Generic (PLEG): container finished" podID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerID="2fdb0e9a9d37e9cf10b5c5be96b2ba5ec7c52098f1884fef22016050028cdca1" exitCode=137
Jul 29 19:10:57.486494 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.486432    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" event={"ID":"ade8b322-0eea-4969-8866-71c0dd7b265e","Type":"ContainerDied","Data":"2fdb0e9a9d37e9cf10b5c5be96b2ba5ec7c52098f1884fef22016050028cdca1"}
Jul 29 19:10:57.486494 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.486441    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"
Jul 29 19:10:57.486574 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.486535    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk" event={"ID":"ade8b322-0eea-4969-8866-71c0dd7b265e","Type":"ContainerDied","Data":"7c7d74d20ba27d7c98c0b038a6eedd27421209799b4ff6cf7ea0459418a1d03c"}
Jul 29 19:10:57.486618 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.486573    2560 scope.go:117] "RemoveContainer" containerID="2fdb0e9a9d37e9cf10b5c5be96b2ba5ec7c52098f1884fef22016050028cdca1"
Jul 29 19:10:57.497284 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.497247    2560 scope.go:117] "RemoveContainer" containerID="a90fbe4f89d3f0acdbb18cda2c574e25cf9aa26484429ee8c1994743fa88f928"
Jul 29 19:10:57.511491 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.511469    2560 scope.go:117] "RemoveContainer" containerID="2fdb0e9a9d37e9cf10b5c5be96b2ba5ec7c52098f1884fef22016050028cdca1"
Jul 29 19:10:57.511857 ip-10-0-137-52 kubenswrapper[2560]: E0729 19:10:57.511831    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fdb0e9a9d37e9cf10b5c5be96b2ba5ec7c52098f1884fef22016050028cdca1\": container with ID starting with 2fdb0e9a9d37e9cf10b5c5be96b2ba5ec7c52098f1884fef22016050028cdca1 not found: ID does not exist" containerID="2fdb0e9a9d37e9cf10b5c5be96b2ba5ec7c52098f1884fef22016050028cdca1"
Jul 29 19:10:57.511971 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.511871    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fdb0e9a9d37e9cf10b5c5be96b2ba5ec7c52098f1884fef22016050028cdca1"} err="failed to get container status \"2fdb0e9a9d37e9cf10b5c5be96b2ba5ec7c52098f1884fef22016050028cdca1\": rpc error: code = NotFound desc = could not find container \"2fdb0e9a9d37e9cf10b5c5be96b2ba5ec7c52098f1884fef22016050028cdca1\": container with ID starting with 2fdb0e9a9d37e9cf10b5c5be96b2ba5ec7c52098f1884fef22016050028cdca1 not found: ID does not exist"
Jul 29 19:10:57.511971 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.511898    2560 scope.go:117] "RemoveContainer" containerID="a90fbe4f89d3f0acdbb18cda2c574e25cf9aa26484429ee8c1994743fa88f928"
Jul 29 19:10:57.512221 ip-10-0-137-52 kubenswrapper[2560]: E0729 19:10:57.512179    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90fbe4f89d3f0acdbb18cda2c574e25cf9aa26484429ee8c1994743fa88f928\": container with ID starting with a90fbe4f89d3f0acdbb18cda2c574e25cf9aa26484429ee8c1994743fa88f928 not found: ID does not exist" containerID="a90fbe4f89d3f0acdbb18cda2c574e25cf9aa26484429ee8c1994743fa88f928"
Jul 29 19:10:57.512295 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.512233    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90fbe4f89d3f0acdbb18cda2c574e25cf9aa26484429ee8c1994743fa88f928"} err="failed to get container status \"a90fbe4f89d3f0acdbb18cda2c574e25cf9aa26484429ee8c1994743fa88f928\": rpc error: code = NotFound desc = could not find container \"a90fbe4f89d3f0acdbb18cda2c574e25cf9aa26484429ee8c1994743fa88f928\": container with ID starting with a90fbe4f89d3f0acdbb18cda2c574e25cf9aa26484429ee8c1994743fa88f928 not found: ID does not exist"
Jul 29 19:10:57.518878 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.518852    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"]
Jul 29 19:10:57.522218 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:57.522183    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7c5b94f5c4-kjmqk"]
Jul 29 19:10:58.690346 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:10:58.690307    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" path="/var/lib/kubelet/pods/ade8b322-0eea-4969-8866-71c0dd7b265e/volumes"
Jul 29 19:11:02.044572 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.044538    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"]
Jul 29 19:11:02.045157 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.045137    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main"
Jul 29 19:11:02.045265 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.045159    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main"
Jul 29 19:11:02.045265 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.045216    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="storage-initializer"
Jul 29 19:11:02.045265 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.045227    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="storage-initializer"
Jul 29 19:11:02.045422 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.045369    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade8b322-0eea-4969-8866-71c0dd7b265e" containerName="main"
Jul 29 19:11:02.079827 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.079795    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"]
Jul 29 19:11:02.079988 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.079945    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.083881 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.083851    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\""
Jul 29 19:11:02.092938 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.092908    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:11:02.151424 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.151373    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.151597 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.151443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.151597 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.151494    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gfvz\" (UniqueName: \"kubernetes.io/projected/25bc0021-c96e-403e-b18a-61a8c472b2fb-kube-api-access-2gfvz\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.151673 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.151642    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.151709 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.151681    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.151709 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.151699    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.151773 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.151755    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25bc0021-c96e-403e-b18a-61a8c472b2fb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.252894 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.252851    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.253096 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.252915    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.253096 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.252942    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.253096 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.253001    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25bc0021-c96e-403e-b18a-61a8c472b2fb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.253096 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.253035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.253096 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.253071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.253410 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.253111    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gfvz\" (UniqueName: \"kubernetes.io/projected/25bc0021-c96e-403e-b18a-61a8c472b2fb-kube-api-access-2gfvz\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.253471 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.253453    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.253613 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.253579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.253613 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.253598    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.253802 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.253696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.255490 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.255463    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.255712 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.255693    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25bc0021-c96e-403e-b18a-61a8c472b2fb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.261847 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.261821    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gfvz\" (UniqueName: \"kubernetes.io/projected/25bc0021-c96e-403e-b18a-61a8c472b2fb-kube-api-access-2gfvz\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.392475 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.392422    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:02.580726 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:02.580695    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"]
Jul 29 19:11:02.582577 ip-10-0-137-52 kubenswrapper[2560]: W0729 19:11:02.582539    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bc0021_c96e_403e_b18a_61a8c472b2fb.slice/crio-d50ffe70ae916821550fc4b4e979d4920e1e4e70bea412bed5cc80b845e598f6 WatchSource:0}: Error finding container d50ffe70ae916821550fc4b4e979d4920e1e4e70bea412bed5cc80b845e598f6: Status 404 returned error can't find the container with id d50ffe70ae916821550fc4b4e979d4920e1e4e70bea412bed5cc80b845e598f6
Jul 29 19:11:03.534798 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:03.534753    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" event={"ID":"25bc0021-c96e-403e-b18a-61a8c472b2fb","Type":"ContainerStarted","Data":"a3e049879a2d009844a8c51e27c054cbe784285847caa3180009d3e9feacf62c"}
Jul 29 19:11:03.535351 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:03.534823    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" event={"ID":"25bc0021-c96e-403e-b18a-61a8c472b2fb","Type":"ContainerStarted","Data":"d50ffe70ae916821550fc4b4e979d4920e1e4e70bea412bed5cc80b845e598f6"}
Jul 29 19:11:05.444729 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:05.444686    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:11:05.450293 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:05.450253    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:11:09.582370 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:09.582334    2560 generic.go:356] "Generic (PLEG): container finished" podID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerID="a3e049879a2d009844a8c51e27c054cbe784285847caa3180009d3e9feacf62c" exitCode=0
Jul 29 19:11:09.582823 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:09.582382    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" event={"ID":"25bc0021-c96e-403e-b18a-61a8c472b2fb","Type":"ContainerDied","Data":"a3e049879a2d009844a8c51e27c054cbe784285847caa3180009d3e9feacf62c"}
Jul 29 19:11:10.598207 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:10.598144    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" event={"ID":"25bc0021-c96e-403e-b18a-61a8c472b2fb","Type":"ContainerStarted","Data":"121f9567fa544e937715830e1edf8b94a4de888c8fc3bef7c03e4dbc13de2468"}
Jul 29 19:11:10.623378 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:10.623314    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podStartSLOduration=2.650576073 podStartE2EDuration="8.623293309s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.972717236s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:10.621557153 +0000 UTC m=+2196.515557555" watchObservedRunningTime="2026-07-29 19:11:10.623293309 +0000 UTC m=+2196.517293713"
Jul 29 19:11:12.393013 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:12.392959    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:12.393547 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:12.393037    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:11:12.395033 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:12.394997    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:11:12.897308 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:12.897215    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:11:15.444767 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:15.444721    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:11:15.450221 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:15.450151    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:11:18.159438 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.159351    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="llm-d-routing-sidecar" containerID="cri-o://dd20029937fc6149a9dba77d72f9d6fe3b53c23e49d071cc4036d6358f125e4d" gracePeriod=2
Jul 29 19:11:18.661209 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.661157    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7_d430a332-a0fb-4a6a-bce7-9cc0d1b21208/main/0.log"
Jul 29 19:11:18.662503 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.662297    2560 generic.go:356] "Generic (PLEG): container finished" podID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerID="b40c8efe347eb3f3054b1e26caaf49e63282aa38bcac20ba10e7172a2eeba953" exitCode=137
Jul 29 19:11:18.662503 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.662325    2560 generic.go:356] "Generic (PLEG): container finished" podID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerID="dd20029937fc6149a9dba77d72f9d6fe3b53c23e49d071cc4036d6358f125e4d" exitCode=0
Jul 29 19:11:18.662704 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.662560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" event={"ID":"d430a332-a0fb-4a6a-bce7-9cc0d1b21208","Type":"ContainerDied","Data":"b40c8efe347eb3f3054b1e26caaf49e63282aa38bcac20ba10e7172a2eeba953"}
Jul 29 19:11:18.662704 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.662657    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" event={"ID":"d430a332-a0fb-4a6a-bce7-9cc0d1b21208","Type":"ContainerDied","Data":"dd20029937fc6149a9dba77d72f9d6fe3b53c23e49d071cc4036d6358f125e4d"}
Jul 29 19:11:18.663503 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.663226    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 19:11:18.664874 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.664847    2560 generic.go:356] "Generic (PLEG): container finished" podID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerID="79dc79d490ce5604df97f11fc602d40fbb79f66ae3f76626007c48c85618e7d3" exitCode=137
Jul 29 19:11:18.664971 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.664891    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" event={"ID":"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9","Type":"ContainerDied","Data":"79dc79d490ce5604df97f11fc602d40fbb79f66ae3f76626007c48c85618e7d3"}
Jul 29 19:11:18.664971 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.664941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg" event={"ID":"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9","Type":"ContainerDied","Data":"614cb709c42fb5a7ab7ebf01993a6d82d6704aebc05ef8c953aff2de7a055dc0"}
Jul 29 19:11:18.665114 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.664972    2560 scope.go:117] "RemoveContainer" containerID="79dc79d490ce5604df97f11fc602d40fbb79f66ae3f76626007c48c85618e7d3"
Jul 29 19:11:18.678368 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.678334    2560 scope.go:117] "RemoveContainer" containerID="cb7941eb390bbe892d184f3306c26907556ae59ed5541a60bb94e0233a68a831"
Jul 29 19:11:18.679452 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.679433    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7_d430a332-a0fb-4a6a-bce7-9cc0d1b21208/main/0.log"
Jul 29 19:11:18.699957 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.699872    2560 scope.go:117] "RemoveContainer" containerID="79dc79d490ce5604df97f11fc602d40fbb79f66ae3f76626007c48c85618e7d3"
Jul 29 19:11:18.700372 ip-10-0-137-52 kubenswrapper[2560]: E0729 19:11:18.700335    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79dc79d490ce5604df97f11fc602d40fbb79f66ae3f76626007c48c85618e7d3\": container with ID starting with 79dc79d490ce5604df97f11fc602d40fbb79f66ae3f76626007c48c85618e7d3 not found: ID does not exist" containerID="79dc79d490ce5604df97f11fc602d40fbb79f66ae3f76626007c48c85618e7d3"
Jul 29 19:11:18.700486 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.700372    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79dc79d490ce5604df97f11fc602d40fbb79f66ae3f76626007c48c85618e7d3"} err="failed to get container status \"79dc79d490ce5604df97f11fc602d40fbb79f66ae3f76626007c48c85618e7d3\": rpc error: code = NotFound desc = could not find container \"79dc79d490ce5604df97f11fc602d40fbb79f66ae3f76626007c48c85618e7d3\": container with ID starting with 79dc79d490ce5604df97f11fc602d40fbb79f66ae3f76626007c48c85618e7d3 not found: ID does not exist"
Jul 29 19:11:18.700486 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.700397    2560 scope.go:117] "RemoveContainer" containerID="cb7941eb390bbe892d184f3306c26907556ae59ed5541a60bb94e0233a68a831"
Jul 29 19:11:18.700767 ip-10-0-137-52 kubenswrapper[2560]: E0729 19:11:18.700743    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7941eb390bbe892d184f3306c26907556ae59ed5541a60bb94e0233a68a831\": container with ID starting with cb7941eb390bbe892d184f3306c26907556ae59ed5541a60bb94e0233a68a831 not found: ID does not exist" containerID="cb7941eb390bbe892d184f3306c26907556ae59ed5541a60bb94e0233a68a831"
Jul 29 19:11:18.700962 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.700933    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7941eb390bbe892d184f3306c26907556ae59ed5541a60bb94e0233a68a831"} err="failed to get container status \"cb7941eb390bbe892d184f3306c26907556ae59ed5541a60bb94e0233a68a831\": rpc error: code = NotFound desc = could not find container \"cb7941eb390bbe892d184f3306c26907556ae59ed5541a60bb94e0233a68a831\": container with ID starting with cb7941eb390bbe892d184f3306c26907556ae59ed5541a60bb94e0233a68a831 not found: ID does not exist"
Jul 29 19:11:18.708349 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.708316    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 19:11:18.723553 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.723464    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-tmp-dir\") pod \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") "
Jul 29 19:11:18.723553 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.723511    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-dshm\") pod \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") "
Jul 29 19:11:18.723553 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.723536    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-tls-certs\") pod \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") "
Jul 29 19:11:18.723824 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.723576    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-kserve-provision-location\") pod \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") "
Jul 29 19:11:18.723824 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.723601    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-tls-certs\") pod \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") "
Jul 29 19:11:18.723824 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.723631    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-model-cache\") pod \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") "
Jul 29 19:11:18.723824 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.723659    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npnr4\" (UniqueName: \"kubernetes.io/projected/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-kube-api-access-npnr4\") pod \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") "
Jul 29 19:11:18.723824 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.723682    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-model-cache\") pod \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") "
Jul 29 19:11:18.723824 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.723751    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-home\") pod \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") "
Jul 29 19:11:18.723824 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.723796    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-kserve-provision-location\") pod \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") "
Jul 29 19:11:18.724220 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.724073    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-model-cache" (OuterVolumeSpecName: "model-cache") pod "d430a332-a0fb-4a6a-bce7-9cc0d1b21208" (UID: "d430a332-a0fb-4a6a-bce7-9cc0d1b21208"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:11:18.724220 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.724088    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" (UID: "1ff6225f-5f30-4745-a2d8-65d4ef07a5f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:11:18.724220 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.724149    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-dshm\") pod \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") "
Jul 29 19:11:18.730273 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.724461    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-home\") pod \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\" (UID: \"1ff6225f-5f30-4745-a2d8-65d4ef07a5f9\") "
Jul 29 19:11:18.730273 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.724497    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thn77\" (UniqueName: \"kubernetes.io/projected/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-kube-api-access-thn77\") pod \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") "
Jul 29 19:11:18.730273 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.724525    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-tmp-dir\") pod \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\" (UID: \"d430a332-a0fb-4a6a-bce7-9cc0d1b21208\") "
Jul 29 19:11:18.730273 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.724827    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:11:18.730273 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.724845    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:11:18.730273 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.725911    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-home" (OuterVolumeSpecName: "home") pod "d430a332-a0fb-4a6a-bce7-9cc0d1b21208" (UID: "d430a332-a0fb-4a6a-bce7-9cc0d1b21208"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:11:18.730273 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.728558    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-home" (OuterVolumeSpecName: "home") pod "1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" (UID: "1ff6225f-5f30-4745-a2d8-65d4ef07a5f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:11:18.732945 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.731224    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" (UID: "1ff6225f-5f30-4745-a2d8-65d4ef07a5f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:11:18.733495 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.733443    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-dshm" (OuterVolumeSpecName: "dshm") pod "d430a332-a0fb-4a6a-bce7-9cc0d1b21208" (UID: "d430a332-a0fb-4a6a-bce7-9cc0d1b21208"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:11:18.735459 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.735413    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-kube-api-access-thn77" (OuterVolumeSpecName: "kube-api-access-thn77") pod "d430a332-a0fb-4a6a-bce7-9cc0d1b21208" (UID: "d430a332-a0fb-4a6a-bce7-9cc0d1b21208"). InnerVolumeSpecName "kube-api-access-thn77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:11:18.748233 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.745265    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" (UID: "1ff6225f-5f30-4745-a2d8-65d4ef07a5f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:11:18.748233 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.745368    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-dshm" (OuterVolumeSpecName: "dshm") pod "1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" (UID: "1ff6225f-5f30-4745-a2d8-65d4ef07a5f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:11:18.748233 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.745426    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d430a332-a0fb-4a6a-bce7-9cc0d1b21208" (UID: "d430a332-a0fb-4a6a-bce7-9cc0d1b21208"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:11:18.750164 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.750130    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" (UID: "1ff6225f-5f30-4745-a2d8-65d4ef07a5f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:11:18.750807 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.750778    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-kube-api-access-npnr4" (OuterVolumeSpecName: "kube-api-access-npnr4") pod "1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" (UID: "1ff6225f-5f30-4745-a2d8-65d4ef07a5f9"). InnerVolumeSpecName "kube-api-access-npnr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:11:18.752852 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.752818    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d430a332-a0fb-4a6a-bce7-9cc0d1b21208" (UID: "d430a332-a0fb-4a6a-bce7-9cc0d1b21208"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:11:18.755721 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.755688    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d430a332-a0fb-4a6a-bce7-9cc0d1b21208" (UID: "d430a332-a0fb-4a6a-bce7-9cc0d1b21208"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:11:18.825506 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.825454    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:11:18.825506 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.825501    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thn77\" (UniqueName: \"kubernetes.io/projected/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-kube-api-access-thn77\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:11:18.825741 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.825549    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:11:18.825741 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.825564    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:11:18.825741 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.825578    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:11:18.825741 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.825589    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:11:18.825741 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.825603    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:11:18.825741 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.825615    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:11:18.825741 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.825631    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npnr4\" (UniqueName: \"kubernetes.io/projected/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-kube-api-access-npnr4\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:11:18.825741 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.825643    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:11:18.825741 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.825657    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:11:18.825741 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:18.825670    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430a332-a0fb-4a6a-bce7-9cc0d1b21208-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:11:19.672705 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:19.672674    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"
Jul 29 19:11:19.674640 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:19.674597    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7_d430a332-a0fb-4a6a-bce7-9cc0d1b21208/main/0.log"
Jul 29 19:11:19.675572 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:19.675548    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"
Jul 29 19:11:19.675572 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:19.675553    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7" event={"ID":"d430a332-a0fb-4a6a-bce7-9cc0d1b21208","Type":"ContainerDied","Data":"0dc3ea0599521cb6edf0db8d70978453b515c06004586aeb58c5cfaa1d49b9bb"}
Jul 29 19:11:19.675755 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:19.675633    2560 scope.go:117] "RemoveContainer" containerID="b40c8efe347eb3f3054b1e26caaf49e63282aa38bcac20ba10e7172a2eeba953"
Jul 29 19:11:19.688595 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:19.688573    2560 scope.go:117] "RemoveContainer" containerID="c41909a05ecd4371fdd0fb41417eeea6bbdeda27af5200062c083a94465301a7"
Jul 29 19:11:19.701450 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:19.701422    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"]
Jul 29 19:11:19.706166 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:19.706136    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57cc8dfc5-kjrcg"]
Jul 29 19:11:19.717520 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:19.717491    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"]
Jul 29 19:11:19.721481 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:19.721095    2560 scope.go:117] "RemoveContainer" containerID="dd20029937fc6149a9dba77d72f9d6fe3b53c23e49d071cc4036d6358f125e4d"
Jul 29 19:11:19.724145 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:19.724120    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6d5c7d975b-2mbz7"]
Jul 29 19:11:20.690826 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:20.690784    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" path="/var/lib/kubelet/pods/1ff6225f-5f30-4745-a2d8-65d4ef07a5f9/volumes"
Jul 29 19:11:20.691459 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:20.691436    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" path="/var/lib/kubelet/pods/d430a332-a0fb-4a6a-bce7-9cc0d1b21208/volumes"
Jul 29 19:11:22.393740 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:22.393692    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:11:25.444490 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:25.444441    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:11:25.450332 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:25.450285    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:11:32.393591 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:32.393527    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:11:35.444552 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:35.444483    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:11:35.450129 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:35.450093    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:11:42.393427 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:42.393376    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:11:45.444851 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:45.444789    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:11:45.450878 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:45.450829    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:11:52.393355 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:52.393312    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:11:55.444867 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:55.444803    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:11:55.450230 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:11:55.450167    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:12:02.393569 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:02.393456    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:12:05.444620 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:05.444565    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:12:05.450107 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:05.450071    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:12:12.393572 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:12.393532    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:12:15.444889 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:15.444843    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:12:15.450238 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:15.450185    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:12:22.393546 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:22.393498    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:12:25.444246 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:25.444175    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:12:25.450566 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:25.450520    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:12:32.393843 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:32.393793    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:12:35.444959 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:35.444910    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:12:35.450539 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:35.450487    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:12:42.393462 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:42.393414    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:12:45.444235 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:45.444166    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:12:45.450523 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:45.450474    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:12:52.393191 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:52.393151    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:12:55.444531 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:55.444487    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:12:55.450631 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:12:55.450589    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:13:02.393954 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:02.393897    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:13:05.444336 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:05.444293    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:13:05.450101 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:05.450058    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:13:12.393520 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:12.393478    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:13:15.444944 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:15.444889    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:13:15.450238 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:15.450186    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:13:22.393355 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:22.393306    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:13:25.444611 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:25.444566    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:13:25.450288 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:25.450248    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:13:32.393486 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:32.393378    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:13:35.444436 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:35.444391    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 29 19:13:35.450405 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:35.450373    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 19:13:42.393339 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:42.393281    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:13:45.460189 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:45.460154    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:13:45.461788 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:45.461756    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:13:45.469347 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:45.469320    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:13:45.472509 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:45.472479    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:13:52.393475 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:13:52.393427    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 19:14:00.322075 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:00.322036    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"]
Jul 29 19:14:00.322949 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:00.322420    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main" containerID="cri-o://0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1" gracePeriod=30
Jul 29 19:14:00.329625 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:00.329598    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"]
Jul 29 19:14:00.329837 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:00.329801    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main" containerID="cri-o://2240c0d54af8fe9ba73cd247c6548708b3d233ddba34aaf00dd526aa402b2ed4" gracePeriod=30
Jul 29 19:14:00.364550 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:00.364503    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:14:02.332406 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:02.332359    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:14:02.406874 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:02.406837    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:14:02.414826 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:02.414792    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:14:13.182742 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.182697    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"]
Jul 29 19:14:13.183131 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.183067    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="llm-d-routing-sidecar"
Jul 29 19:14:13.183131 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.183078    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="llm-d-routing-sidecar"
Jul 29 19:14:13.183131 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.183099    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main"
Jul 29 19:14:13.183131 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.183105    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main"
Jul 29 19:14:13.183131 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.183115    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main"
Jul 29 19:14:13.183131 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.183120    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main"
Jul 29 19:14:13.183131 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.183130    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="storage-initializer"
Jul 29 19:14:13.183131 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.183135    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="storage-initializer"
Jul 29 19:14:13.183402 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.183147    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="storage-initializer"
Jul 29 19:14:13.183402 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.183153    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="storage-initializer"
Jul 29 19:14:13.183402 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.183225    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="llm-d-routing-sidecar"
Jul 29 19:14:13.183402 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.183235    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ff6225f-5f30-4745-a2d8-65d4ef07a5f9" containerName="main"
Jul 29 19:14:13.183402 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.183246    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d430a332-a0fb-4a6a-bce7-9cc0d1b21208" containerName="main"
Jul 29 19:14:13.185496 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.185480    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.188266 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.188245    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 29 19:14:13.188266 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.188246    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-zcqkf\""
Jul 29 19:14:13.198795 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.198770    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"]
Jul 29 19:14:13.243643 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.243610    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.243799 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.243662    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4535dee7-acb6-4874-bdc3-8ec5d8046ded-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.243799 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.243716    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-dshm\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.243799 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.243769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-home\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.243967 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.243804    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-model-cache\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.243967 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.243844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5lck\" (UniqueName: \"kubernetes.io/projected/4535dee7-acb6-4874-bdc3-8ec5d8046ded-kube-api-access-c5lck\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.244051 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.243974    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.248099 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.248068    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"]
Jul 29 19:14:13.250764 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.250742    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.263657 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.263631    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"]
Jul 29 19:14:13.275920 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.275893    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:14:13.345326 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345294    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-model-cache\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.345534 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345335    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lck\" (UniqueName: \"kubernetes.io/projected/4535dee7-acb6-4874-bdc3-8ec5d8046ded-kube-api-access-c5lck\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.345534 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345365    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.345534 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.345534 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345448    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.345534 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345504    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90746bf6-fbba-4be7-a826-64f5e5bb4785-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.345814 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345538    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.345814 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345587    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.345814 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345627    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4535dee7-acb6-4874-bdc3-8ec5d8046ded-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.345814 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345647    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlpd7\" (UniqueName: \"kubernetes.io/projected/90746bf6-fbba-4be7-a826-64f5e5bb4785-kube-api-access-hlpd7\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.345814 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345673    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-dshm\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.345814 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345689    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.345814 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345702    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-model-cache\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.346215 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345854    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.346215 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345862    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.346215 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345890    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.346215 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.345936    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-home\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.346377 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.346238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-home\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.347886 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.347868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-dshm\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.348286 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.348266    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4535dee7-acb6-4874-bdc3-8ec5d8046ded-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.354584 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.354561    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5lck\" (UniqueName: \"kubernetes.io/projected/4535dee7-acb6-4874-bdc3-8ec5d8046ded-kube-api-access-c5lck\") pod \"custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.447099 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.447013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.447099 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.447063    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.447323 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.447109    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.447323 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.447127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.447323 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.447158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.447323 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.447228    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90746bf6-fbba-4be7-a826-64f5e5bb4785-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.447323 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.447282    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlpd7\" (UniqueName: \"kubernetes.io/projected/90746bf6-fbba-4be7-a826-64f5e5bb4785-kube-api-access-hlpd7\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.447596 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.447520    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.447596 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.447566    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.447710 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.447657    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.447710 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.447685    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.449398 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.449377    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.449775 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.449753    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90746bf6-fbba-4be7-a826-64f5e5bb4785-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.457454 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.457427    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlpd7\" (UniqueName: \"kubernetes.io/projected/90746bf6-fbba-4be7-a826-64f5e5bb4785-kube-api-access-hlpd7\") pod \"custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.494856 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.494817    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.510634 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.510604    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:14:13.561643 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.561610    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:13.650323 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.648245    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"]
Jul 29 19:14:13.708834 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.708797    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"]
Jul 29 19:14:13.711091 ip-10-0-137-52 kubenswrapper[2560]: W0729 19:14:13.711055    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90746bf6_fbba_4be7_a826_64f5e5bb4785.slice/crio-53d7336d60a321cf092c80a037494b31bed5f3eb2967f51726d798777be29fd3 WatchSource:0}: Error finding container 53d7336d60a321cf092c80a037494b31bed5f3eb2967f51726d798777be29fd3: Status 404 returned error can't find the container with id 53d7336d60a321cf092c80a037494b31bed5f3eb2967f51726d798777be29fd3
Jul 29 19:14:13.752655 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.752622    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" event={"ID":"90746bf6-fbba-4be7-a826-64f5e5bb4785","Type":"ContainerStarted","Data":"53d7336d60a321cf092c80a037494b31bed5f3eb2967f51726d798777be29fd3"}
Jul 29 19:14:13.754089 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.754059    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" event={"ID":"4535dee7-acb6-4874-bdc3-8ec5d8046ded","Type":"ContainerStarted","Data":"d25ce0195653b892b2b0d3d3682d30ee6eb6f136db13aa141e028f513fc6acbf"}
Jul 29 19:14:13.754209 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.754109    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" event={"ID":"4535dee7-acb6-4874-bdc3-8ec5d8046ded","Type":"ContainerStarted","Data":"d698fc19f9e08e39d2d2ced0a641c72cadfb1dfc8461d84f8ce19725db1e57f0"}
Jul 29 19:14:13.754209 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.754141    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:13.885538 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:13.885497    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:14:14.765158 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:14.765108    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" event={"ID":"90746bf6-fbba-4be7-a826-64f5e5bb4785","Type":"ContainerStarted","Data":"30545e391b2d5692a8b996c834a120a7ff2b7525c49190fee2ac7b43693fac22"}
Jul 29 19:14:14.768030 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:14.768003    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" event={"ID":"4535dee7-acb6-4874-bdc3-8ec5d8046ded","Type":"ContainerStarted","Data":"ca32ba3e9ae05ab07a8e900fc036386f438958a65727899a7e8005277fdaf904"}
Jul 29 19:14:19.808478 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:19.808417    2560 generic.go:356] "Generic (PLEG): container finished" podID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerID="ca32ba3e9ae05ab07a8e900fc036386f438958a65727899a7e8005277fdaf904" exitCode=0
Jul 29 19:14:19.809330 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:19.808631    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" event={"ID":"4535dee7-acb6-4874-bdc3-8ec5d8046ded","Type":"ContainerDied","Data":"ca32ba3e9ae05ab07a8e900fc036386f438958a65727899a7e8005277fdaf904"}
Jul 29 19:14:20.817637 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:20.817602    2560 generic.go:356] "Generic (PLEG): container finished" podID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerID="30545e391b2d5692a8b996c834a120a7ff2b7525c49190fee2ac7b43693fac22" exitCode=0
Jul 29 19:14:20.818173 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:20.817685    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" event={"ID":"90746bf6-fbba-4be7-a826-64f5e5bb4785","Type":"ContainerDied","Data":"30545e391b2d5692a8b996c834a120a7ff2b7525c49190fee2ac7b43693fac22"}
Jul 29 19:14:20.820450 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:20.820421    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" event={"ID":"4535dee7-acb6-4874-bdc3-8ec5d8046ded","Type":"ContainerStarted","Data":"ab0f82759c1658c9882cd3a3b7092e49e15658a6132764ed696e853f9f92729c"}
Jul 29 19:14:20.870358 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:20.870308    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podStartSLOduration=1.991119683 podStartE2EDuration="7.870291194s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.879171511s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:14:20.869253989 +0000 UTC m=+2386.763254409" watchObservedRunningTime="2026-07-29 19:14:20.870291194 +0000 UTC m=+2386.764291596"
Jul 29 19:14:21.830065 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:21.830029    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" event={"ID":"90746bf6-fbba-4be7-a826-64f5e5bb4785","Type":"ContainerStarted","Data":"6afdad50c06072147a84fe7c92e51622039289cfcb380fa136016fbc886ea706"}
Jul 29 19:14:21.850923 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:21.850856    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podStartSLOduration=2.281493152 podStartE2EDuration="8.850834344s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.569341192s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:14:21.850505883 +0000 UTC m=+2387.744506308" watchObservedRunningTime="2026-07-29 19:14:21.850834344 +0000 UTC m=+2387.744834746"
Jul 29 19:14:23.495727 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:23.495685    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:23.496487 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:23.495763    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:23.497583 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:23.497535    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:14:23.510495 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:23.510475    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:14:23.562254 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:23.562218    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:23.562447 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:23.562276    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:14:23.563957 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:23.563921    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:14:30.323063 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.322995    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="llm-d-routing-sidecar" containerID="cri-o://e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de" gracePeriod=2
Jul 29 19:14:30.673675 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.673645    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg_8d93eef3-4de0-4a8d-ada0-cc477c5c50b2/main/0.log"
Jul 29 19:14:30.686771 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.686734    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:14:30.706433 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.706405    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:14:30.827283 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827175    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-home\") pod \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") "
Jul 29 19:14:30.827283 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827230    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-home\") pod \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") "
Jul 29 19:14:30.827283 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827271    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-kserve-provision-location\") pod \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") "
Jul 29 19:14:30.827577 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827291    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-tmp-dir\") pod \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") "
Jul 29 19:14:30.827577 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827315    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-dshm\") pod \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") "
Jul 29 19:14:30.827577 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827333    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-tls-certs\") pod \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") "
Jul 29 19:14:30.827577 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827365    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tq4v\" (UniqueName: \"kubernetes.io/projected/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-kube-api-access-5tq4v\") pod \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") "
Jul 29 19:14:30.827577 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827422    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-dshm\") pod \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") "
Jul 29 19:14:30.827577 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827471    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-model-cache\") pod \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") "
Jul 29 19:14:30.827879 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827593    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-tmp-dir\") pod \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") "
Jul 29 19:14:30.827879 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827638    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff86q\" (UniqueName: \"kubernetes.io/projected/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-kube-api-access-ff86q\") pod \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") "
Jul 29 19:14:30.827879 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827680    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-model-cache\") pod \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") "
Jul 29 19:14:30.827879 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827712    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-kserve-provision-location\") pod \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\" (UID: \"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d\") "
Jul 29 19:14:30.827879 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827735    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-tls-certs\") pod \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\" (UID: \"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2\") "
Jul 29 19:14:30.828134 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.827936    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-home" (OuterVolumeSpecName: "home") pod "7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" (UID: "7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:14:30.828134 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.828123    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:14:30.828277 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.828233    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-home" (OuterVolumeSpecName: "home") pod "8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" (UID: "8d93eef3-4de0-4a8d-ada0-cc477c5c50b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:14:30.831961 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.830447    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" (UID: "8d93eef3-4de0-4a8d-ada0-cc477c5c50b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:14:30.833219 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.833175    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" (UID: "7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:14:30.833448 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.833428    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-dshm" (OuterVolumeSpecName: "dshm") pod "8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" (UID: "8d93eef3-4de0-4a8d-ada0-cc477c5c50b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:14:30.833768 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.833748    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" (UID: "7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:14:30.833969 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.833950    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-kube-api-access-ff86q" (OuterVolumeSpecName: "kube-api-access-ff86q") pod "8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" (UID: "8d93eef3-4de0-4a8d-ada0-cc477c5c50b2"). InnerVolumeSpecName "kube-api-access-ff86q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:14:30.834141 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.834123    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-kube-api-access-5tq4v" (OuterVolumeSpecName: "kube-api-access-5tq4v") pod "7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" (UID: "7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d"). InnerVolumeSpecName "kube-api-access-5tq4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:14:30.837356 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.837338    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-dshm" (OuterVolumeSpecName: "dshm") pod "7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" (UID: "7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:14:30.846311 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.846266    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" (UID: "7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:14:30.849790 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.847438    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" (UID: "8d93eef3-4de0-4a8d-ada0-cc477c5c50b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:14:30.849790 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.848663    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" (UID: "8d93eef3-4de0-4a8d-ada0-cc477c5c50b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:14:30.875927 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.875883    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" (UID: "8d93eef3-4de0-4a8d-ada0-cc477c5c50b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:14:30.889277 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.889237    2560 generic.go:356] "Generic (PLEG): container finished" podID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerID="2240c0d54af8fe9ba73cd247c6548708b3d233ddba34aaf00dd526aa402b2ed4" exitCode=137
Jul 29 19:14:30.889462 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.889297    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"
Jul 29 19:14:30.889462 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.889345    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" event={"ID":"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d","Type":"ContainerDied","Data":"2240c0d54af8fe9ba73cd247c6548708b3d233ddba34aaf00dd526aa402b2ed4"}
Jul 29 19:14:30.889462 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.889405    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q" event={"ID":"7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d","Type":"ContainerDied","Data":"0a1e7af5797177f640160d6b3e7b306cf6bf3556c44919e72026e9f88a9165a5"}
Jul 29 19:14:30.889462 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.889448    2560 scope.go:117] "RemoveContainer" containerID="2240c0d54af8fe9ba73cd247c6548708b3d233ddba34aaf00dd526aa402b2ed4"
Jul 29 19:14:30.891221 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.891182    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg_8d93eef3-4de0-4a8d-ada0-cc477c5c50b2/main/0.log"
Jul 29 19:14:30.891986 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.891961    2560 generic.go:356] "Generic (PLEG): container finished" podID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerID="0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1" exitCode=137
Jul 29 19:14:30.891986 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.891985    2560 generic.go:356] "Generic (PLEG): container finished" podID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerID="e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de" exitCode=0
Jul 29 19:14:30.892173 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.892021    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"
Jul 29 19:14:30.892173 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.892032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" event={"ID":"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2","Type":"ContainerDied","Data":"0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1"}
Jul 29 19:14:30.892173 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.892078    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" event={"ID":"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2","Type":"ContainerDied","Data":"e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de"}
Jul 29 19:14:30.892173 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.892106    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg" event={"ID":"8d93eef3-4de0-4a8d-ada0-cc477c5c50b2","Type":"ContainerDied","Data":"021e1175404a3d7981fbce305f6898c4d8c68022775f30f8564ef3cb4b7e5452"}
Jul 29 19:14:30.901255 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.901213    2560 scope.go:117] "RemoveContainer" containerID="067a415ecf7952aa79a7c41b133b9c171c34a4ec0fe5637c697e0c94fe56de52"
Jul 29 19:14:30.914706 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.914664    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" (UID: "7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:14:30.919903 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.919875    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"]
Jul 29 19:14:30.926585 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.926558    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-54697dd99c6xgqg"]
Jul 29 19:14:30.929263 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.929241    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff86q\" (UniqueName: \"kubernetes.io/projected/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-kube-api-access-ff86q\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:14:30.929342 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.929269    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:14:30.929342 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.929288    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:14:30.929342 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.929302    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:14:30.929342 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.929317    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:14:30.929342 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.929329    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:14:30.929342 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.929341    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:14:30.929581 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.929353    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:14:30.929581 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.929365    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:14:30.929581 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.929378    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tq4v\" (UniqueName: \"kubernetes.io/projected/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-kube-api-access-5tq4v\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:14:30.929581 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.929390    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:14:30.929581 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.929402    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:14:30.929581 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.929414    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:14:30.957392 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.957363    2560 scope.go:117] "RemoveContainer" containerID="2240c0d54af8fe9ba73cd247c6548708b3d233ddba34aaf00dd526aa402b2ed4"
Jul 29 19:14:30.957743 ip-10-0-137-52 kubenswrapper[2560]: E0729 19:14:30.957716    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2240c0d54af8fe9ba73cd247c6548708b3d233ddba34aaf00dd526aa402b2ed4\": container with ID starting with 2240c0d54af8fe9ba73cd247c6548708b3d233ddba34aaf00dd526aa402b2ed4 not found: ID does not exist" containerID="2240c0d54af8fe9ba73cd247c6548708b3d233ddba34aaf00dd526aa402b2ed4"
Jul 29 19:14:30.957834 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.957756    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2240c0d54af8fe9ba73cd247c6548708b3d233ddba34aaf00dd526aa402b2ed4"} err="failed to get container status \"2240c0d54af8fe9ba73cd247c6548708b3d233ddba34aaf00dd526aa402b2ed4\": rpc error: code = NotFound desc = could not find container \"2240c0d54af8fe9ba73cd247c6548708b3d233ddba34aaf00dd526aa402b2ed4\": container with ID starting with 2240c0d54af8fe9ba73cd247c6548708b3d233ddba34aaf00dd526aa402b2ed4 not found: ID does not exist"
Jul 29 19:14:30.957834 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.957780    2560 scope.go:117] "RemoveContainer" containerID="067a415ecf7952aa79a7c41b133b9c171c34a4ec0fe5637c697e0c94fe56de52"
Jul 29 19:14:30.958073 ip-10-0-137-52 kubenswrapper[2560]: E0729 19:14:30.958051    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"067a415ecf7952aa79a7c41b133b9c171c34a4ec0fe5637c697e0c94fe56de52\": container with ID starting with 067a415ecf7952aa79a7c41b133b9c171c34a4ec0fe5637c697e0c94fe56de52 not found: ID does not exist" containerID="067a415ecf7952aa79a7c41b133b9c171c34a4ec0fe5637c697e0c94fe56de52"
Jul 29 19:14:30.958154 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.958084    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"067a415ecf7952aa79a7c41b133b9c171c34a4ec0fe5637c697e0c94fe56de52"} err="failed to get container status \"067a415ecf7952aa79a7c41b133b9c171c34a4ec0fe5637c697e0c94fe56de52\": rpc error: code = NotFound desc = could not find container \"067a415ecf7952aa79a7c41b133b9c171c34a4ec0fe5637c697e0c94fe56de52\": container with ID starting with 067a415ecf7952aa79a7c41b133b9c171c34a4ec0fe5637c697e0c94fe56de52 not found: ID does not exist"
Jul 29 19:14:30.958154 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.958109    2560 scope.go:117] "RemoveContainer" containerID="0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1"
Jul 29 19:14:30.967973 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:30.967954    2560 scope.go:117] "RemoveContainer" containerID="1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29"
Jul 29 19:14:31.005914 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.005889    2560 scope.go:117] "RemoveContainer" containerID="e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de"
Jul 29 19:14:31.016733 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.016709    2560 scope.go:117] "RemoveContainer" containerID="0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1"
Jul 29 19:14:31.017063 ip-10-0-137-52 kubenswrapper[2560]: E0729 19:14:31.017036    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1\": container with ID starting with 0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1 not found: ID does not exist" containerID="0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1"
Jul 29 19:14:31.017137 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.017076    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1"} err="failed to get container status \"0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1\": rpc error: code = NotFound desc = could not find container \"0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1\": container with ID starting with 0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1 not found: ID does not exist"
Jul 29 19:14:31.017137 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.017106    2560 scope.go:117] "RemoveContainer" containerID="1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29"
Jul 29 19:14:31.017460 ip-10-0-137-52 kubenswrapper[2560]: E0729 19:14:31.017435    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29\": container with ID starting with 1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29 not found: ID does not exist" containerID="1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29"
Jul 29 19:14:31.017594 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.017464    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29"} err="failed to get container status \"1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29\": rpc error: code = NotFound desc = could not find container \"1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29\": container with ID starting with 1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29 not found: ID does not exist"
Jul 29 19:14:31.017594 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.017492    2560 scope.go:117] "RemoveContainer" containerID="e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de"
Jul 29 19:14:31.017762 ip-10-0-137-52 kubenswrapper[2560]: E0729 19:14:31.017740    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de\": container with ID starting with e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de not found: ID does not exist" containerID="e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de"
Jul 29 19:14:31.017827 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.017766    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de"} err="failed to get container status \"e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de\": rpc error: code = NotFound desc = could not find container \"e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de\": container with ID starting with e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de not found: ID does not exist"
Jul 29 19:14:31.017827 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.017786    2560 scope.go:117] "RemoveContainer" containerID="0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1"
Jul 29 19:14:31.018010 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.017991    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1"} err="failed to get container status \"0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1\": rpc error: code = NotFound desc = could not find container \"0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1\": container with ID starting with 0639b1c18b749ddce96807d382144604d19d4fe3754a47b025b44c0e04b34ca1 not found: ID does not exist"
Jul 29 19:14:31.018010 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.018008    2560 scope.go:117] "RemoveContainer" containerID="1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29"
Jul 29 19:14:31.018235 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.018213    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29"} err="failed to get container status \"1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29\": rpc error: code = NotFound desc = could not find container \"1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29\": container with ID starting with 1b64b5ece6eb8ea9c5f2ec034f027bcdc937adfb23a8051f39293c49e9e3dc29 not found: ID does not exist"
Jul 29 19:14:31.018235 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.018236    2560 scope.go:117] "RemoveContainer" containerID="e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de"
Jul 29 19:14:31.018441 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.018422    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de"} err="failed to get container status \"e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de\": rpc error: code = NotFound desc = could not find container \"e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de\": container with ID starting with e65654ea56164d927b7ca602329eb9fa394d6b15334296368fddeb5c4bb7a3de not found: ID does not exist"
Jul 29 19:14:31.214893 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.214862    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"]
Jul 29 19:14:31.218746 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:31.218705    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-b9zpw6q"]
Jul 29 19:14:32.691071 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:32.691027    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" path="/var/lib/kubelet/pods/7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d/volumes"
Jul 29 19:14:32.691745 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:32.691719    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" path="/var/lib/kubelet/pods/8d93eef3-4de0-4a8d-ada0-cc477c5c50b2/volumes"
Jul 29 19:14:33.496342 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:33.496267    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:14:33.562808 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:33.562770    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:14:34.733258 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:34.733225    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 19:14:34.739363 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:34.739339    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 19:14:43.495672 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:43.495605    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:14:43.562451 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:43.562402    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:14:49.881335 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:49.881299    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:14:50.740562 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:50.740531    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"]
Jul 29 19:14:50.740792 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:50.740760    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main" containerID="cri-o://121f9567fa544e937715830e1edf8b94a4de888c8fc3bef7c03e4dbc13de2468" gracePeriod=30
Jul 29 19:14:50.932812 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:50.932773    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:14:53.496174 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:53.496125    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:14:53.562952 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:14:53.562897    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:15:03.496408 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:03.496306    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:15:03.562982 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:03.562934    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:15:06.884756 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:06.884710    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:15:07.090926 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.090888    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:15:07.380474 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.380437    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:15:07.443719 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.443671    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 29 19:15:07.444313 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.444287    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="llm-d-routing-sidecar"
Jul 29 19:15:07.444404 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.444316    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="llm-d-routing-sidecar"
Jul 29 19:15:07.444404 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.444330    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main"
Jul 29 19:15:07.444404 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.444340    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main"
Jul 29 19:15:07.444404 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.444363    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="storage-initializer"
Jul 29 19:15:07.444404 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.444374    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="storage-initializer"
Jul 29 19:15:07.444404 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.444386    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="storage-initializer"
Jul 29 19:15:07.444404 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.444394    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="storage-initializer"
Jul 29 19:15:07.444637 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.444424    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main"
Jul 29 19:15:07.444637 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.444432    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main"
Jul 29 19:15:07.444637 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.444532    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e25e2f1-d4d0-47c5-a8e8-67c0c4a7de0d" containerName="main"
Jul 29 19:15:07.444637 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.444546    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="main"
Jul 29 19:15:07.444637 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.444555    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d93eef3-4de0-4a8d-ada0-cc477c5c50b2" containerName="llm-d-routing-sidecar"
Jul 29 19:15:07.447789 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.447769    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.452251 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.452225    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-dm5z2\""
Jul 29 19:15:07.452692 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.452666    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
Jul 29 19:15:07.461402 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.461376    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 29 19:15:07.470740 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.470706    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.470910 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.470750    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.470910 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.470796    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.470910 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.470812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09a6da2b-1451-4d8c-8c3c-020bfccb6497-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.470910 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.470829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.470910 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.470861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr4dg\" (UniqueName: \"kubernetes.io/projected/09a6da2b-1451-4d8c-8c3c-020bfccb6497-kube-api-access-kr4dg\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.471221 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.470917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.495255 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.495221    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:15:07.571787 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.571748    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.571997 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.571818    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.571997 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.571859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.571997 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.571922    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.572251 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.572228    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.572348 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.572301    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.572406 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.572362    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09a6da2b-1451-4d8c-8c3c-020bfccb6497-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.572460 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.572399    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.572460 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.572442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr4dg\" (UniqueName: \"kubernetes.io/projected/09a6da2b-1451-4d8c-8c3c-020bfccb6497-kube-api-access-kr4dg\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.572590 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.572565    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.572941 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.572907    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.574715 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.574669    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.575665 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.575642    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09a6da2b-1451-4d8c-8c3c-020bfccb6497-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.589868 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.589830    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr4dg\" (UniqueName: \"kubernetes.io/projected/09a6da2b-1451-4d8c-8c3c-020bfccb6497-kube-api-access-kr4dg\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.761943 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.761854    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:07.926310 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:07.926282    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 29 19:15:07.927534 ip-10-0-137-52 kubenswrapper[2560]: W0729 19:15:07.927506    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a6da2b_1451_4d8c_8c3c_020bfccb6497.slice/crio-541f56a9252cc9dcf75ec7bf39ea72e0a0464cba287bb789da6ad96c4624780d WatchSource:0}: Error finding container 541f56a9252cc9dcf75ec7bf39ea72e0a0464cba287bb789da6ad96c4624780d: Status 404 returned error can't find the container with id 541f56a9252cc9dcf75ec7bf39ea72e0a0464cba287bb789da6ad96c4624780d
Jul 29 19:15:08.052859 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:08.052744    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:15:08.122282 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:08.122234    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"09a6da2b-1451-4d8c-8c3c-020bfccb6497","Type":"ContainerStarted","Data":"460f469e01c005f2b59ea55b043eacad3b17ea1b3f2e2691dddb4ba2eb04cc05"}
Jul 29 19:15:08.122493 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:08.122312    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"09a6da2b-1451-4d8c-8c3c-020bfccb6497","Type":"ContainerStarted","Data":"541f56a9252cc9dcf75ec7bf39ea72e0a0464cba287bb789da6ad96c4624780d"}
Jul 29 19:15:13.495223 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:13.495151    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:15:13.562698 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:13.562656    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:15:14.166628 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:14.166587    2560 generic.go:356] "Generic (PLEG): container finished" podID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerID="460f469e01c005f2b59ea55b043eacad3b17ea1b3f2e2691dddb4ba2eb04cc05" exitCode=0
Jul 29 19:15:14.166808 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:14.166663    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"09a6da2b-1451-4d8c-8c3c-020bfccb6497","Type":"ContainerDied","Data":"460f469e01c005f2b59ea55b043eacad3b17ea1b3f2e2691dddb4ba2eb04cc05"}
Jul 29 19:15:15.176147 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:15.176102    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"09a6da2b-1451-4d8c-8c3c-020bfccb6497","Type":"ContainerStarted","Data":"a72992e40241005e46dc51b0a3f6b965d9cae1161d5e3ca878ee4ac8a2becc08"}
Jul 29 19:15:15.199979 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:15.199906    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=2.123528117 podStartE2EDuration="8.199883361s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.076355244s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:15:15.197030057 +0000 UTC m=+2441.091030479" watchObservedRunningTime="2026-07-29 19:15:15.199883361 +0000 UTC m=+2441.093883763"
Jul 29 19:15:17.763031 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:17.762977    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:17.764835 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:17.764807    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 29 19:15:21.192419 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.192394    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv_25bc0021-c96e-403e-b18a-61a8c472b2fb/main/0.log"
Jul 29 19:15:21.210723 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.210697    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:15:21.218493 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.218464    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv_25bc0021-c96e-403e-b18a-61a8c472b2fb/main/0.log"
Jul 29 19:15:21.218885 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.218857    2560 generic.go:356] "Generic (PLEG): container finished" podID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerID="121f9567fa544e937715830e1edf8b94a4de888c8fc3bef7c03e4dbc13de2468" exitCode=137
Jul 29 19:15:21.219031 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.218925    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"
Jul 29 19:15:21.219031 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.218937    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" event={"ID":"25bc0021-c96e-403e-b18a-61a8c472b2fb","Type":"ContainerDied","Data":"121f9567fa544e937715830e1edf8b94a4de888c8fc3bef7c03e4dbc13de2468"}
Jul 29 19:15:21.219031 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.219000    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv" event={"ID":"25bc0021-c96e-403e-b18a-61a8c472b2fb","Type":"ContainerDied","Data":"d50ffe70ae916821550fc4b4e979d4920e1e4e70bea412bed5cc80b845e598f6"}
Jul 29 19:15:21.219031 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.219024    2560 scope.go:117] "RemoveContainer" containerID="121f9567fa544e937715830e1edf8b94a4de888c8fc3bef7c03e4dbc13de2468"
Jul 29 19:15:21.228792 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.228763    2560 scope.go:117] "RemoveContainer" containerID="a3e049879a2d009844a8c51e27c054cbe784285847caa3180009d3e9feacf62c"
Jul 29 19:15:21.257893 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.257869    2560 scope.go:117] "RemoveContainer" containerID="121f9567fa544e937715830e1edf8b94a4de888c8fc3bef7c03e4dbc13de2468"
Jul 29 19:15:21.262839 ip-10-0-137-52 kubenswrapper[2560]: E0729 19:15:21.260705    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121f9567fa544e937715830e1edf8b94a4de888c8fc3bef7c03e4dbc13de2468\": container with ID starting with 121f9567fa544e937715830e1edf8b94a4de888c8fc3bef7c03e4dbc13de2468 not found: ID does not exist" containerID="121f9567fa544e937715830e1edf8b94a4de888c8fc3bef7c03e4dbc13de2468"
Jul 29 19:15:21.262839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.260748    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121f9567fa544e937715830e1edf8b94a4de888c8fc3bef7c03e4dbc13de2468"} err="failed to get container status \"121f9567fa544e937715830e1edf8b94a4de888c8fc3bef7c03e4dbc13de2468\": rpc error: code = NotFound desc = could not find container \"121f9567fa544e937715830e1edf8b94a4de888c8fc3bef7c03e4dbc13de2468\": container with ID starting with 121f9567fa544e937715830e1edf8b94a4de888c8fc3bef7c03e4dbc13de2468 not found: ID does not exist"
Jul 29 19:15:21.262839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.260773    2560 scope.go:117] "RemoveContainer" containerID="a3e049879a2d009844a8c51e27c054cbe784285847caa3180009d3e9feacf62c"
Jul 29 19:15:21.262839 ip-10-0-137-52 kubenswrapper[2560]: E0729 19:15:21.261330    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e049879a2d009844a8c51e27c054cbe784285847caa3180009d3e9feacf62c\": container with ID starting with a3e049879a2d009844a8c51e27c054cbe784285847caa3180009d3e9feacf62c not found: ID does not exist" containerID="a3e049879a2d009844a8c51e27c054cbe784285847caa3180009d3e9feacf62c"
Jul 29 19:15:21.262839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.261365    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e049879a2d009844a8c51e27c054cbe784285847caa3180009d3e9feacf62c"} err="failed to get container status \"a3e049879a2d009844a8c51e27c054cbe784285847caa3180009d3e9feacf62c\": rpc error: code = NotFound desc = could not find container \"a3e049879a2d009844a8c51e27c054cbe784285847caa3180009d3e9feacf62c\": container with ID starting with a3e049879a2d009844a8c51e27c054cbe784285847caa3180009d3e9feacf62c not found: ID does not exist"
Jul 29 19:15:21.310262 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.310231    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25bc0021-c96e-403e-b18a-61a8c472b2fb-tls-certs\") pod \"25bc0021-c96e-403e-b18a-61a8c472b2fb\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") "
Jul 29 19:15:21.310460 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.310295    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-kserve-provision-location\") pod \"25bc0021-c96e-403e-b18a-61a8c472b2fb\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") "
Jul 29 19:15:21.310460 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.310337    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-dshm\") pod \"25bc0021-c96e-403e-b18a-61a8c472b2fb\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") "
Jul 29 19:15:21.310460 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.310368    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-tmp-dir\") pod \"25bc0021-c96e-403e-b18a-61a8c472b2fb\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") "
Jul 29 19:15:21.310460 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.310406    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-home\") pod \"25bc0021-c96e-403e-b18a-61a8c472b2fb\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") "
Jul 29 19:15:21.310460 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.310432    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-model-cache\") pod \"25bc0021-c96e-403e-b18a-61a8c472b2fb\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") "
Jul 29 19:15:21.312576 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.310523    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gfvz\" (UniqueName: \"kubernetes.io/projected/25bc0021-c96e-403e-b18a-61a8c472b2fb-kube-api-access-2gfvz\") pod \"25bc0021-c96e-403e-b18a-61a8c472b2fb\" (UID: \"25bc0021-c96e-403e-b18a-61a8c472b2fb\") "
Jul 29 19:15:21.312576 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.310885    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "25bc0021-c96e-403e-b18a-61a8c472b2fb" (UID: "25bc0021-c96e-403e-b18a-61a8c472b2fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:15:21.312576 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.312174    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-home" (OuterVolumeSpecName: "home") pod "25bc0021-c96e-403e-b18a-61a8c472b2fb" (UID: "25bc0021-c96e-403e-b18a-61a8c472b2fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:15:21.313140 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.313093    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-dshm" (OuterVolumeSpecName: "dshm") pod "25bc0021-c96e-403e-b18a-61a8c472b2fb" (UID: "25bc0021-c96e-403e-b18a-61a8c472b2fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:15:21.313506 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.313478    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25bc0021-c96e-403e-b18a-61a8c472b2fb-kube-api-access-2gfvz" (OuterVolumeSpecName: "kube-api-access-2gfvz") pod "25bc0021-c96e-403e-b18a-61a8c472b2fb" (UID: "25bc0021-c96e-403e-b18a-61a8c472b2fb"). InnerVolumeSpecName "kube-api-access-2gfvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:15:21.314582 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.314556    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25bc0021-c96e-403e-b18a-61a8c472b2fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25bc0021-c96e-403e-b18a-61a8c472b2fb" (UID: "25bc0021-c96e-403e-b18a-61a8c472b2fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:15:21.331879 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.331849    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25bc0021-c96e-403e-b18a-61a8c472b2fb" (UID: "25bc0021-c96e-403e-b18a-61a8c472b2fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:15:21.340251 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.340220    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "25bc0021-c96e-403e-b18a-61a8c472b2fb" (UID: "25bc0021-c96e-403e-b18a-61a8c472b2fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:15:21.412246 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.412212    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:15:21.412246 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.412245    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gfvz\" (UniqueName: \"kubernetes.io/projected/25bc0021-c96e-403e-b18a-61a8c472b2fb-kube-api-access-2gfvz\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:15:21.412477 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.412259    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25bc0021-c96e-403e-b18a-61a8c472b2fb-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:15:21.412477 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.412274    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:15:21.412477 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.412286    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:15:21.412477 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.412297    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:15:21.412477 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.412308    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25bc0021-c96e-403e-b18a-61a8c472b2fb-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:15:21.548466 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.548432    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"]
Jul 29 19:15:21.554758 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:21.554716    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75f4d95578vtmbv"]
Jul 29 19:15:22.689826 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:22.689790    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" path="/var/lib/kubelet/pods/25bc0021-c96e-403e-b18a-61a8c472b2fb/volumes"
Jul 29 19:15:23.495473 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:23.495423    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:15:23.562493 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:23.562437    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:15:25.202706 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:25.202663    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:15:27.763218 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:27.763141    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 29 19:15:33.495445 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:33.495379    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:15:33.562938 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:33.562885    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:15:37.762835 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:37.762793    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:15:37.763391 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:37.763243    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 29 19:15:43.496308 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:43.496140    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:15:43.563008 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:43.562970    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:15:47.762869 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:47.762817    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 29 19:15:53.496004 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:53.495955    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:15:53.563014 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:53.562953    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:15:57.762964 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:15:57.762910    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 29 19:16:03.496107 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:03.496055    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:16:03.562317 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:03.562272    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:16:07.762361 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:07.762311    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 29 19:16:13.495432 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:13.495380    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:16:13.562404 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:13.562358    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:16:17.762981 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:17.762932    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 29 19:16:23.495835 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:23.495789    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:16:23.562467 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:23.562425    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:16:27.763348 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:27.763254    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 29 19:16:33.495915 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:33.495858    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:16:33.563043 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:33.563001    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:16:37.762895 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:37.762855    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 29 19:16:43.495752 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:43.495705    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Jul 29 19:16:43.562479 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:43.562435    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 19:16:47.763003 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:47.762962    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 29 19:16:53.507077 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:53.507046    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:16:53.525051 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:53.525026    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:16:53.572763 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:53.572731    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:16:53.581472 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:53.581441    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:16:57.762604 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:16:57.762554    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 29 19:17:07.762662 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:07.762616    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 29 19:17:11.831331 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:11.831296    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"]
Jul 29 19:17:11.831819 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:11.831677    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main" containerID="cri-o://ab0f82759c1658c9882cd3a3b7092e49e15658a6132764ed696e853f9f92729c" gracePeriod=30
Jul 29 19:17:11.836881 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:11.836850    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"]
Jul 29 19:17:11.837069 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:11.837043    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main" containerID="cri-o://6afdad50c06072147a84fe7c92e51622039289cfcb380fa136016fbc886ea706" gracePeriod=30
Jul 29 19:17:11.861842 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:11.861805    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:17:11.931171 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:11.931134    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:17:17.762646 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:17.762600    2560 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 29 19:17:27.772744 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:27.772709    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:17:27.780319 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:27.780291    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:17:35.564157 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:35.564127    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:17:35.626517 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:35.626486    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 29 19:17:35.626713 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:35.626693    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main" containerID="cri-o://a72992e40241005e46dc51b0a3f6b965d9cae1161d5e3ca878ee4ac8a2becc08" gracePeriod=30
Jul 29 19:17:35.681842 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:35.681791    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:17:36.381757 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.381733    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:17:36.470724 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.470585    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-model-cache\") pod \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") "
Jul 29 19:17:36.470724 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.470620    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr4dg\" (UniqueName: \"kubernetes.io/projected/09a6da2b-1451-4d8c-8c3c-020bfccb6497-kube-api-access-kr4dg\") pod \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") "
Jul 29 19:17:36.470724 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.470640    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-home\") pod \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") "
Jul 29 19:17:36.470724 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.470660    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-kserve-provision-location\") pod \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") "
Jul 29 19:17:36.471067 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.470739    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09a6da2b-1451-4d8c-8c3c-020bfccb6497-tls-certs\") pod \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") "
Jul 29 19:17:36.471067 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.470783    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-dshm\") pod \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") "
Jul 29 19:17:36.471067 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.470806    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-tmp-dir\") pod \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\" (UID: \"09a6da2b-1451-4d8c-8c3c-020bfccb6497\") "
Jul 29 19:17:36.471067 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.470907    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-model-cache" (OuterVolumeSpecName: "model-cache") pod "09a6da2b-1451-4d8c-8c3c-020bfccb6497" (UID: "09a6da2b-1451-4d8c-8c3c-020bfccb6497"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:36.471327 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.471096    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:36.471395 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.471369    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-home" (OuterVolumeSpecName: "home") pod "09a6da2b-1451-4d8c-8c3c-020bfccb6497" (UID: "09a6da2b-1451-4d8c-8c3c-020bfccb6497"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:36.472906 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.472875    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-dshm" (OuterVolumeSpecName: "dshm") pod "09a6da2b-1451-4d8c-8c3c-020bfccb6497" (UID: "09a6da2b-1451-4d8c-8c3c-020bfccb6497"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:36.473225 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.473176    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a6da2b-1451-4d8c-8c3c-020bfccb6497-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09a6da2b-1451-4d8c-8c3c-020bfccb6497" (UID: "09a6da2b-1451-4d8c-8c3c-020bfccb6497"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:17:36.473225 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.473216    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a6da2b-1451-4d8c-8c3c-020bfccb6497-kube-api-access-kr4dg" (OuterVolumeSpecName: "kube-api-access-kr4dg") pod "09a6da2b-1451-4d8c-8c3c-020bfccb6497" (UID: "09a6da2b-1451-4d8c-8c3c-020bfccb6497"). InnerVolumeSpecName "kube-api-access-kr4dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:17:36.487420 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.487394    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09a6da2b-1451-4d8c-8c3c-020bfccb6497" (UID: "09a6da2b-1451-4d8c-8c3c-020bfccb6497"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:36.536028 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.535970    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09a6da2b-1451-4d8c-8c3c-020bfccb6497" (UID: "09a6da2b-1451-4d8c-8c3c-020bfccb6497"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:36.572019 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.571991    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09a6da2b-1451-4d8c-8c3c-020bfccb6497-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:36.572019 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.572015    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:36.572019 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.572024    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:36.572432 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.572033    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr4dg\" (UniqueName: \"kubernetes.io/projected/09a6da2b-1451-4d8c-8c3c-020bfccb6497-kube-api-access-kr4dg\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:36.572432 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.572042    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:36.572432 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:36.572051    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a6da2b-1451-4d8c-8c3c-020bfccb6497-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:37.050117 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:37.050082    2560 generic.go:356] "Generic (PLEG): container finished" podID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerID="a72992e40241005e46dc51b0a3f6b965d9cae1161d5e3ca878ee4ac8a2becc08" exitCode=0
Jul 29 19:17:37.050322 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:37.050135    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 19:17:37.050322 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:37.050173    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"09a6da2b-1451-4d8c-8c3c-020bfccb6497","Type":"ContainerDied","Data":"a72992e40241005e46dc51b0a3f6b965d9cae1161d5e3ca878ee4ac8a2becc08"}
Jul 29 19:17:37.050322 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:37.050255    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"09a6da2b-1451-4d8c-8c3c-020bfccb6497","Type":"ContainerDied","Data":"541f56a9252cc9dcf75ec7bf39ea72e0a0464cba287bb789da6ad96c4624780d"}
Jul 29 19:17:37.050322 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:37.050287    2560 scope.go:117] "RemoveContainer" containerID="a72992e40241005e46dc51b0a3f6b965d9cae1161d5e3ca878ee4ac8a2becc08"
Jul 29 19:17:37.059465 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:37.059444    2560 scope.go:117] "RemoveContainer" containerID="460f469e01c005f2b59ea55b043eacad3b17ea1b3f2e2691dddb4ba2eb04cc05"
Jul 29 19:17:37.075893 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:37.075858    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 29 19:17:37.081875 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:37.081848    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 29 19:17:37.130057 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:37.130030    2560 scope.go:117] "RemoveContainer" containerID="a72992e40241005e46dc51b0a3f6b965d9cae1161d5e3ca878ee4ac8a2becc08"
Jul 29 19:17:37.130430 ip-10-0-137-52 kubenswrapper[2560]: E0729 19:17:37.130404    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72992e40241005e46dc51b0a3f6b965d9cae1161d5e3ca878ee4ac8a2becc08\": container with ID starting with a72992e40241005e46dc51b0a3f6b965d9cae1161d5e3ca878ee4ac8a2becc08 not found: ID does not exist" containerID="a72992e40241005e46dc51b0a3f6b965d9cae1161d5e3ca878ee4ac8a2becc08"
Jul 29 19:17:37.130535 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:37.130439    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72992e40241005e46dc51b0a3f6b965d9cae1161d5e3ca878ee4ac8a2becc08"} err="failed to get container status \"a72992e40241005e46dc51b0a3f6b965d9cae1161d5e3ca878ee4ac8a2becc08\": rpc error: code = NotFound desc = could not find container \"a72992e40241005e46dc51b0a3f6b965d9cae1161d5e3ca878ee4ac8a2becc08\": container with ID starting with a72992e40241005e46dc51b0a3f6b965d9cae1161d5e3ca878ee4ac8a2becc08 not found: ID does not exist"
Jul 29 19:17:37.130535 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:37.130459    2560 scope.go:117] "RemoveContainer" containerID="460f469e01c005f2b59ea55b043eacad3b17ea1b3f2e2691dddb4ba2eb04cc05"
Jul 29 19:17:37.130745 ip-10-0-137-52 kubenswrapper[2560]: E0729 19:17:37.130722    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460f469e01c005f2b59ea55b043eacad3b17ea1b3f2e2691dddb4ba2eb04cc05\": container with ID starting with 460f469e01c005f2b59ea55b043eacad3b17ea1b3f2e2691dddb4ba2eb04cc05 not found: ID does not exist" containerID="460f469e01c005f2b59ea55b043eacad3b17ea1b3f2e2691dddb4ba2eb04cc05"
Jul 29 19:17:37.130821 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:37.130756    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460f469e01c005f2b59ea55b043eacad3b17ea1b3f2e2691dddb4ba2eb04cc05"} err="failed to get container status \"460f469e01c005f2b59ea55b043eacad3b17ea1b3f2e2691dddb4ba2eb04cc05\": rpc error: code = NotFound desc = could not find container \"460f469e01c005f2b59ea55b043eacad3b17ea1b3f2e2691dddb4ba2eb04cc05\": container with ID starting with 460f469e01c005f2b59ea55b043eacad3b17ea1b3f2e2691dddb4ba2eb04cc05 not found: ID does not exist"
Jul 29 19:17:37.166788 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:37.166750    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:17:37.318723 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:37.318689    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:17:38.687368 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:38.687338    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" path="/var/lib/kubelet/pods/09a6da2b-1451-4d8c-8c3c-020bfccb6497/volumes"
Jul 29 19:17:41.831843 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:41.831800    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="llm-d-routing-sidecar" containerID="cri-o://d25ce0195653b892b2b0d3d3682d30ee6eb6f136db13aa141e028f513fc6acbf" gracePeriod=2
Jul 29 19:17:42.079308 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.079149    2560 generic.go:356] "Generic (PLEG): container finished" podID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerID="6afdad50c06072147a84fe7c92e51622039289cfcb380fa136016fbc886ea706" exitCode=137
Jul 29 19:17:42.079308 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.079261    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" event={"ID":"90746bf6-fbba-4be7-a826-64f5e5bb4785","Type":"ContainerDied","Data":"6afdad50c06072147a84fe7c92e51622039289cfcb380fa136016fbc886ea706"}
Jul 29 19:17:42.082303 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.082239    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4_4535dee7-acb6-4874-bdc3-8ec5d8046ded/main/0.log"
Jul 29 19:17:42.083161 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.083137    2560 generic.go:356] "Generic (PLEG): container finished" podID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerID="ab0f82759c1658c9882cd3a3b7092e49e15658a6132764ed696e853f9f92729c" exitCode=137
Jul 29 19:17:42.083161 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.083159    2560 generic.go:356] "Generic (PLEG): container finished" podID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerID="d25ce0195653b892b2b0d3d3682d30ee6eb6f136db13aa141e028f513fc6acbf" exitCode=0
Jul 29 19:17:42.083345 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.083212    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" event={"ID":"4535dee7-acb6-4874-bdc3-8ec5d8046ded","Type":"ContainerDied","Data":"ab0f82759c1658c9882cd3a3b7092e49e15658a6132764ed696e853f9f92729c"}
Jul 29 19:17:42.083345 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.083254    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" event={"ID":"4535dee7-acb6-4874-bdc3-8ec5d8046ded","Type":"ContainerDied","Data":"d25ce0195653b892b2b0d3d3682d30ee6eb6f136db13aa141e028f513fc6acbf"}
Jul 29 19:17:42.115624 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.115598    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4_4535dee7-acb6-4874-bdc3-8ec5d8046ded/main/0.log"
Jul 29 19:17:42.118592 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.118571    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:17:42.136667 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.136640    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:17:42.225997 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.225970    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90746bf6-fbba-4be7-a826-64f5e5bb4785-tls-certs\") pod \"90746bf6-fbba-4be7-a826-64f5e5bb4785\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") "
Jul 29 19:17:42.225997 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226002    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-model-cache\") pod \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") "
Jul 29 19:17:42.226267 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226026    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-kserve-provision-location\") pod \"90746bf6-fbba-4be7-a826-64f5e5bb4785\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") "
Jul 29 19:17:42.226267 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226064    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-kserve-provision-location\") pod \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") "
Jul 29 19:17:42.226267 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226078    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4535dee7-acb6-4874-bdc3-8ec5d8046ded-tls-certs\") pod \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") "
Jul 29 19:17:42.226267 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226104    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-home\") pod \"90746bf6-fbba-4be7-a826-64f5e5bb4785\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") "
Jul 29 19:17:42.226267 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226127    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5lck\" (UniqueName: \"kubernetes.io/projected/4535dee7-acb6-4874-bdc3-8ec5d8046ded-kube-api-access-c5lck\") pod \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") "
Jul 29 19:17:42.226267 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226150    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-dshm\") pod \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") "
Jul 29 19:17:42.226267 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226175    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-dshm\") pod \"90746bf6-fbba-4be7-a826-64f5e5bb4785\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") "
Jul 29 19:17:42.226267 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226221    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-home\") pod \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") "
Jul 29 19:17:42.226267 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226268    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-tmp-dir\") pod \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\" (UID: \"4535dee7-acb6-4874-bdc3-8ec5d8046ded\") "
Jul 29 19:17:42.226753 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226281    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-model-cache" (OuterVolumeSpecName: "model-cache") pod "4535dee7-acb6-4874-bdc3-8ec5d8046ded" (UID: "4535dee7-acb6-4874-bdc3-8ec5d8046ded"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:42.226753 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226294    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlpd7\" (UniqueName: \"kubernetes.io/projected/90746bf6-fbba-4be7-a826-64f5e5bb4785-kube-api-access-hlpd7\") pod \"90746bf6-fbba-4be7-a826-64f5e5bb4785\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") "
Jul 29 19:17:42.226753 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226340    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-model-cache\") pod \"90746bf6-fbba-4be7-a826-64f5e5bb4785\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") "
Jul 29 19:17:42.226753 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226365    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-tmp-dir\") pod \"90746bf6-fbba-4be7-a826-64f5e5bb4785\" (UID: \"90746bf6-fbba-4be7-a826-64f5e5bb4785\") "
Jul 29 19:17:42.226753 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.226662    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:42.227213 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.227168    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-home" (OuterVolumeSpecName: "home") pod "90746bf6-fbba-4be7-a826-64f5e5bb4785" (UID: "90746bf6-fbba-4be7-a826-64f5e5bb4785"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:42.229116 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.228803    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90746bf6-fbba-4be7-a826-64f5e5bb4785-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90746bf6-fbba-4be7-a826-64f5e5bb4785" (UID: "90746bf6-fbba-4be7-a826-64f5e5bb4785"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:17:42.229116 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.228858    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-dshm" (OuterVolumeSpecName: "dshm") pod "90746bf6-fbba-4be7-a826-64f5e5bb4785" (UID: "90746bf6-fbba-4be7-a826-64f5e5bb4785"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:42.229514 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.229490    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4535dee7-acb6-4874-bdc3-8ec5d8046ded-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4535dee7-acb6-4874-bdc3-8ec5d8046ded" (UID: "4535dee7-acb6-4874-bdc3-8ec5d8046ded"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:17:42.230047 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.229835    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4535dee7-acb6-4874-bdc3-8ec5d8046ded-kube-api-access-c5lck" (OuterVolumeSpecName: "kube-api-access-c5lck") pod "4535dee7-acb6-4874-bdc3-8ec5d8046ded" (UID: "4535dee7-acb6-4874-bdc3-8ec5d8046ded"). InnerVolumeSpecName "kube-api-access-c5lck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:17:42.230874 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.230584    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-home" (OuterVolumeSpecName: "home") pod "4535dee7-acb6-4874-bdc3-8ec5d8046ded" (UID: "4535dee7-acb6-4874-bdc3-8ec5d8046ded"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:42.231104 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.231060    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-model-cache" (OuterVolumeSpecName: "model-cache") pod "90746bf6-fbba-4be7-a826-64f5e5bb4785" (UID: "90746bf6-fbba-4be7-a826-64f5e5bb4785"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:42.231310 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.231283    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-dshm" (OuterVolumeSpecName: "dshm") pod "4535dee7-acb6-4874-bdc3-8ec5d8046ded" (UID: "4535dee7-acb6-4874-bdc3-8ec5d8046ded"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:42.231586 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.231559    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90746bf6-fbba-4be7-a826-64f5e5bb4785-kube-api-access-hlpd7" (OuterVolumeSpecName: "kube-api-access-hlpd7") pod "90746bf6-fbba-4be7-a826-64f5e5bb4785" (UID: "90746bf6-fbba-4be7-a826-64f5e5bb4785"). InnerVolumeSpecName "kube-api-access-hlpd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:17:42.245992 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.245956    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90746bf6-fbba-4be7-a826-64f5e5bb4785" (UID: "90746bf6-fbba-4be7-a826-64f5e5bb4785"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:42.248907 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.248880    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4535dee7-acb6-4874-bdc3-8ec5d8046ded" (UID: "4535dee7-acb6-4874-bdc3-8ec5d8046ded"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:42.296451 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.296417    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4535dee7-acb6-4874-bdc3-8ec5d8046ded" (UID: "4535dee7-acb6-4874-bdc3-8ec5d8046ded"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:42.300435 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.300408    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90746bf6-fbba-4be7-a826-64f5e5bb4785" (UID: "90746bf6-fbba-4be7-a826-64f5e5bb4785"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:17:42.327944 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.327916    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-model-cache\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:42.327944 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.327940    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:42.328090 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.327949    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90746bf6-fbba-4be7-a826-64f5e5bb4785-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:42.328090 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.327958    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:42.328090 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.327967    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-kserve-provision-location\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:42.328090 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.327977    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4535dee7-acb6-4874-bdc3-8ec5d8046ded-tls-certs\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:42.328090 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.327987    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:42.328090 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.327996    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5lck\" (UniqueName: \"kubernetes.io/projected/4535dee7-acb6-4874-bdc3-8ec5d8046ded-kube-api-access-c5lck\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:42.328090 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.328003    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:42.328090 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.328012    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90746bf6-fbba-4be7-a826-64f5e5bb4785-dshm\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:42.328090 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.328019    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-home\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:42.328090 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.328026    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4535dee7-acb6-4874-bdc3-8ec5d8046ded-tmp-dir\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:42.328090 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:42.328034    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlpd7\" (UniqueName: \"kubernetes.io/projected/90746bf6-fbba-4be7-a826-64f5e5bb4785-kube-api-access-hlpd7\") on node \"ip-10-0-137-52.ec2.internal\" DevicePath \"\""
Jul 29 19:17:43.089883 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:43.089852    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"
Jul 29 19:17:43.090339 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:43.089851    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h" event={"ID":"90746bf6-fbba-4be7-a826-64f5e5bb4785","Type":"ContainerDied","Data":"53d7336d60a321cf092c80a037494b31bed5f3eb2967f51726d798777be29fd3"}
Jul 29 19:17:43.090339 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:43.089999    2560 scope.go:117] "RemoveContainer" containerID="6afdad50c06072147a84fe7c92e51622039289cfcb380fa136016fbc886ea706"
Jul 29 19:17:43.091501 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:43.091479    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4_4535dee7-acb6-4874-bdc3-8ec5d8046ded/main/0.log"
Jul 29 19:17:43.092485 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:43.092450    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"
Jul 29 19:17:43.092820 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:43.092445    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4" event={"ID":"4535dee7-acb6-4874-bdc3-8ec5d8046ded","Type":"ContainerDied","Data":"d698fc19f9e08e39d2d2ced0a641c72cadfb1dfc8461d84f8ce19725db1e57f0"}
Jul 29 19:17:43.099342 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:43.099313    2560 scope.go:117] "RemoveContainer" containerID="30545e391b2d5692a8b996c834a120a7ff2b7525c49190fee2ac7b43693fac22"
Jul 29 19:17:43.110177 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:43.110154    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"]
Jul 29 19:17:43.115130 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:43.115110    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7585fcfd79-jx65h"]
Jul 29 19:17:43.132869 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:43.132846    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"]
Jul 29 19:17:43.137870 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:43.137844    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76746f6c44-zmvl4"]
Jul 29 19:17:43.170626 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:43.170598    2560 scope.go:117] "RemoveContainer" containerID="ab0f82759c1658c9882cd3a3b7092e49e15658a6132764ed696e853f9f92729c"
Jul 29 19:17:43.179184 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:43.179163    2560 scope.go:117] "RemoveContainer" containerID="ca32ba3e9ae05ab07a8e900fc036386f438958a65727899a7e8005277fdaf904"
Jul 29 19:17:43.245607 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:43.245577    2560 scope.go:117] "RemoveContainer" containerID="d25ce0195653b892b2b0d3d3682d30ee6eb6f136db13aa141e028f513fc6acbf"
Jul 29 19:17:44.687592 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:44.687555    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" path="/var/lib/kubelet/pods/4535dee7-acb6-4874-bdc3-8ec5d8046ded/volumes"
Jul 29 19:17:44.688228 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:44.688190    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" path="/var/lib/kubelet/pods/90746bf6-fbba-4be7-a826-64f5e5bb4785/volumes"
Jul 29 19:17:54.026612 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:54.026581    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/2.log"
Jul 29 19:17:54.027136 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:54.027078    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/1.log"
Jul 29 19:17:54.040035 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:54.040010    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/1.log"
Jul 29 19:17:54.040802 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:54.040783    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/0.log"
Jul 29 19:17:55.024362 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:55.024333    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/2.log"
Jul 29 19:17:55.024536 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:55.024516    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/1.log"
Jul 29 19:17:55.040261 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:55.040236    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/1.log"
Jul 29 19:17:55.040597 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:55.040415    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/0.log"
Jul 29 19:17:56.054316 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:56.054234    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/2.log"
Jul 29 19:17:56.054811 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:56.054425    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/1.log"
Jul 29 19:17:56.066962 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:56.066935    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/1.log"
Jul 29 19:17:56.067508 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:56.067492    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/0.log"
Jul 29 19:17:57.062837 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:57.062805    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/2.log"
Jul 29 19:17:57.063278 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:57.063042    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/1.log"
Jul 29 19:17:57.075889 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:57.075859    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/1.log"
Jul 29 19:17:57.076268 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:57.076252    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/0.log"
Jul 29 19:17:58.049783 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:58.049755    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/2.log"
Jul 29 19:17:58.050067 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:58.050045    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/1.log"
Jul 29 19:17:58.068098 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:58.068072    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/1.log"
Jul 29 19:17:58.068098 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:58.068102    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/0.log"
Jul 29 19:17:59.031006 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:59.030971    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/2.log"
Jul 29 19:17:59.031302 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:59.031146    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/1.log"
Jul 29 19:17:59.045916 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:59.045887    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/1.log"
Jul 29 19:17:59.046318 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:17:59.046303    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/0.log"
Jul 29 19:18:00.007965 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:00.007934    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/2.log"
Jul 29 19:18:00.008390 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:00.008253    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/1.log"
Jul 29 19:18:00.022105 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:00.022082    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/1.log"
Jul 29 19:18:00.022105 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:00.022097    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/0.log"
Jul 29 19:18:00.987661 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:00.987632    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/2.log"
Jul 29 19:18:00.987838 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:00.987760    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/1.log"
Jul 29 19:18:01.001397 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:01.001374    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/1.log"
Jul 29 19:18:01.001535 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:01.001500    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/0.log"
Jul 29 19:18:01.989900 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:01.989865    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/2.log"
Jul 29 19:18:01.990329 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:01.990143    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/1.log"
Jul 29 19:18:02.004155 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:02.004129    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/1.log"
Jul 29 19:18:02.004414 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:02.004395    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/0.log"
Jul 29 19:18:03.035468 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:03.035436    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/1.log"
Jul 29 19:18:03.036348 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:03.036332    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/2.log"
Jul 29 19:18:03.049655 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:03.049629    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/1.log"
Jul 29 19:18:03.050145 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:03.050129    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/0.log"
Jul 29 19:18:04.054475 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:04.054446    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/2.log"
Jul 29 19:18:04.054869 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:04.054674    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/1.log"
Jul 29 19:18:04.074060 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:04.074035    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/1.log"
Jul 29 19:18:04.074299 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:04.074286    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/0.log"
Jul 29 19:18:05.181826 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:05.181792    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/2.log"
Jul 29 19:18:05.182218 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:05.182043    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/1.log"
Jul 29 19:18:05.204806 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:05.204776    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/1.log"
Jul 29 19:18:05.204951 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:05.204942    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/0.log"
Jul 29 19:18:06.331678 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:06.331649    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/2.log"
Jul 29 19:18:06.332133 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:06.331933    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/1.log"
Jul 29 19:18:06.348333 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:06.348302    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/1.log"
Jul 29 19:18:06.348457 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:06.348401    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/0.log"
Jul 29 19:18:07.453489 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:07.453451    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/2.log"
Jul 29 19:18:07.453964 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:07.453649    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-56f678f445-9wrf5_c31372be-e035-44b2-88c2-85e009edb9fc/istio-proxy/1.log"
Jul 29 19:18:07.468093 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:07.468068    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/1.log"
Jul 29 19:18:07.468349 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:07.468328    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-67f67c9c77-jdf6t_489997c7-eab7-4688-9685-91c2d4f18e32/istio-proxy/0.log"
Jul 29 19:18:08.511289 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:08.511258    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-w95c8_d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2/istio-proxy/1.log"
Jul 29 19:18:08.550994 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:08.550965    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-w95c8_d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2/istio-proxy/0.log"
Jul 29 19:18:09.430460 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:09.430410    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-w95c8_d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2/istio-proxy/0.log"
Jul 29 19:18:09.430957 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:09.430916    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-w95c8_d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2/istio-proxy/1.log"
Jul 29 19:18:10.277097 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:10.277067    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-gdd8w_affc3e91-e990-48f0-93f8-361c8072a238/manager/0.log"
Jul 29 19:18:10.305887 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:10.305858    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-7vlkb_a262ea76-3d0a-4db4-8dcd-8c9ef79419f1/manager/0.log"
Jul 29 19:18:10.392614 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:10.392584    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-47frn_1f129975-2ae4-4482-9ee0-fb04916afecb/limitador/0.log"
Jul 29 19:18:10.409239 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:10.409210    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-8gnc8_877d5c15-68fa-4dad-b51f-ad6c5f154f58/manager/0.log"
Jul 29 19:18:12.627581 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.627548    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:18:12.788111 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788081    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xjw8s/must-gather-g76rl"]
Jul 29 19:18:12.788447 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788434    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main"
Jul 29 19:18:12.788514 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788449    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main"
Jul 29 19:18:12.788514 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788463    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="storage-initializer"
Jul 29 19:18:12.788514 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788471    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="storage-initializer"
Jul 29 19:18:12.788514 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788479    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main"
Jul 29 19:18:12.788514 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788484    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main"
Jul 29 19:18:12.788514 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788493    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="llm-d-routing-sidecar"
Jul 29 19:18:12.788514 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788498    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="llm-d-routing-sidecar"
Jul 29 19:18:12.788514 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788505    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main"
Jul 29 19:18:12.788514 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788510    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main"
Jul 29 19:18:12.788839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788520    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main"
Jul 29 19:18:12.788839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788526    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main"
Jul 29 19:18:12.788839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788534    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="storage-initializer"
Jul 29 19:18:12.788839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788539    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="storage-initializer"
Jul 29 19:18:12.788839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788547    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="storage-initializer"
Jul 29 19:18:12.788839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788552    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="storage-initializer"
Jul 29 19:18:12.788839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788559    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="storage-initializer"
Jul 29 19:18:12.788839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788564    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="storage-initializer"
Jul 29 19:18:12.788839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788637    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="main"
Jul 29 19:18:12.788839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788647    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="90746bf6-fbba-4be7-a826-64f5e5bb4785" containerName="main"
Jul 29 19:18:12.788839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788657    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="25bc0021-c96e-403e-b18a-61a8c472b2fb" containerName="main"
Jul 29 19:18:12.788839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788667    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="09a6da2b-1451-4d8c-8c3c-020bfccb6497" containerName="main"
Jul 29 19:18:12.788839 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.788677    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4535dee7-acb6-4874-bdc3-8ec5d8046ded" containerName="llm-d-routing-sidecar"
Jul 29 19:18:12.791815 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.791796    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xjw8s/must-gather-g76rl"
Jul 29 19:18:12.795166 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.795144    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xjw8s\"/\"default-dockercfg-z2hkg\""
Jul 29 19:18:12.795166 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.795157    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xjw8s\"/\"openshift-service-ca.crt\""
Jul 29 19:18:12.795376 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.795320    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xjw8s\"/\"kube-root-ca.crt\""
Jul 29 19:18:12.807573 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.807547    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xjw8s/must-gather-g76rl"]
Jul 29 19:18:12.834486 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.834453    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:18:12.879328 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.879231    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ffa275bb-4430-4c78-820c-1ef38b13b4fa-must-gather-output\") pod \"must-gather-g76rl\" (UID: \"ffa275bb-4430-4c78-820c-1ef38b13b4fa\") " pod="openshift-must-gather-xjw8s/must-gather-g76rl"
Jul 29 19:18:12.879328 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.879288    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qz6b\" (UniqueName: \"kubernetes.io/projected/ffa275bb-4430-4c78-820c-1ef38b13b4fa-kube-api-access-6qz6b\") pod \"must-gather-g76rl\" (UID: \"ffa275bb-4430-4c78-820c-1ef38b13b4fa\") " pod="openshift-must-gather-xjw8s/must-gather-g76rl"
Jul 29 19:18:12.948860 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.948818    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:18:12.980050 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.980016    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ffa275bb-4430-4c78-820c-1ef38b13b4fa-must-gather-output\") pod \"must-gather-g76rl\" (UID: \"ffa275bb-4430-4c78-820c-1ef38b13b4fa\") " pod="openshift-must-gather-xjw8s/must-gather-g76rl"
Jul 29 19:18:12.980225 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.980082    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qz6b\" (UniqueName: \"kubernetes.io/projected/ffa275bb-4430-4c78-820c-1ef38b13b4fa-kube-api-access-6qz6b\") pod \"must-gather-g76rl\" (UID: \"ffa275bb-4430-4c78-820c-1ef38b13b4fa\") " pod="openshift-must-gather-xjw8s/must-gather-g76rl"
Jul 29 19:18:12.980371 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.980351    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ffa275bb-4430-4c78-820c-1ef38b13b4fa-must-gather-output\") pod \"must-gather-g76rl\" (UID: \"ffa275bb-4430-4c78-820c-1ef38b13b4fa\") " pod="openshift-must-gather-xjw8s/must-gather-g76rl"
Jul 29 19:18:12.996447 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:12.996421    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qz6b\" (UniqueName: \"kubernetes.io/projected/ffa275bb-4430-4c78-820c-1ef38b13b4fa-kube-api-access-6qz6b\") pod \"must-gather-g76rl\" (UID: \"ffa275bb-4430-4c78-820c-1ef38b13b4fa\") " pod="openshift-must-gather-xjw8s/must-gather-g76rl"
Jul 29 19:18:13.100711 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:13.100665    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xjw8s/must-gather-g76rl"
Jul 29 19:18:13.238997 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:13.238970    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xjw8s/must-gather-g76rl"]
Jul 29 19:18:13.241601 ip-10-0-137-52 kubenswrapper[2560]: W0729 19:18:13.241571    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffa275bb_4430_4c78_820c_1ef38b13b4fa.slice/crio-f0ad37486f088a1be4f5f4335f7e56c6004aa248199eda8d93c57169d52ebdeb WatchSource:0}: Error finding container f0ad37486f088a1be4f5f4335f7e56c6004aa248199eda8d93c57169d52ebdeb: Status 404 returned error can't find the container with id f0ad37486f088a1be4f5f4335f7e56c6004aa248199eda8d93c57169d52ebdeb
Jul 29 19:18:13.243586 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:13.243567    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:18:13.249622 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:13.249599    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xjw8s/must-gather-g76rl" event={"ID":"ffa275bb-4430-4c78-820c-1ef38b13b4fa","Type":"ContainerStarted","Data":"f0ad37486f088a1be4f5f4335f7e56c6004aa248199eda8d93c57169d52ebdeb"}
Jul 29 19:18:14.256179 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:14.256152    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xjw8s/must-gather-g76rl" event={"ID":"ffa275bb-4430-4c78-820c-1ef38b13b4fa","Type":"ContainerStarted","Data":"1703eb9ca916caf323b2f374a48c08b93696680e4c1d6e54641789d342631fd7"}
Jul 29 19:18:15.264803 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:15.264767    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xjw8s/must-gather-g76rl" event={"ID":"ffa275bb-4430-4c78-820c-1ef38b13b4fa","Type":"ContainerStarted","Data":"e86c5bdbeadba71f7213abe359d84700a469f2446a622b5c29729f35adcec0f8"}
Jul 29 19:18:15.288210 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:15.288139    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xjw8s/must-gather-g76rl" podStartSLOduration=2.373495783 podStartE2EDuration="3.2881223s" totalImagesPullingTime="914.626517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:15.287176344 +0000 UTC m=+2621.181176745" watchObservedRunningTime="2026-07-29 19:18:15.2881223 +0000 UTC m=+2621.182122700"
Jul 29 19:18:16.048158 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:16.048125    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rrsh7_244afbce-35b6-4432-a874-d7081b20c82e/global-pull-secret-syncer/0.log"
Jul 29 19:18:16.097289 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:16.097258    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log"
Jul 29 19:18:16.373819 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:16.373787    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-52.ec2.internal_25ae268d780129888ee17e93df9f46c1/haproxy/0.log"
Jul 29 19:18:18.783585 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:18.783546    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:18:18.881740 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:18.881701    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d6sh_2c4effb3-c122-47d9-88ad-557670f8248d/konnectivity-agent/0.log" limit=1024
Jul 29 19:18:20.363817 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:20.363780    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-gdd8w_affc3e91-e990-48f0-93f8-361c8072a238/manager/0.log"
Jul 29 19:18:20.406490 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:20.406458    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-7vlkb_a262ea76-3d0a-4db4-8dcd-8c9ef79419f1/manager/0.log"
Jul 29 19:18:20.520997 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:20.520963    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-47frn_1f129975-2ae4-4482-9ee0-fb04916afecb/limitador/0.log"
Jul 29 19:18:20.562299 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:20.562266    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-8gnc8_877d5c15-68fa-4dad-b51f-ad6c5f154f58/manager/0.log"
Jul 29 19:18:22.670585 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:22.670552    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9q7nb_7745c9cd-f92a-44e3-b485-96782dd3de5a/node-exporter/0.log"
Jul 29 19:18:22.712311 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:22.712129    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9q7nb_7745c9cd-f92a-44e3-b485-96782dd3de5a/kube-rbac-proxy/0.log"
Jul 29 19:18:22.740137 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:22.740103    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9q7nb_7745c9cd-f92a-44e3-b485-96782dd3de5a/init-textfile/0.log"
Jul 29 19:18:26.047660 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.047619    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"]
Jul 29 19:18:26.052793 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.052764    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.062525 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.062439    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"]
Jul 29 19:18:26.112675 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.112635    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb3c08b3-377d-4914-8c83-422b0e21ef02-proc\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.112877 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.112682    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njpd2\" (UniqueName: \"kubernetes.io/projected/bb3c08b3-377d-4914-8c83-422b0e21ef02-kube-api-access-njpd2\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.112877 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.112715    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb3c08b3-377d-4914-8c83-422b0e21ef02-lib-modules\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.112877 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.112736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb3c08b3-377d-4914-8c83-422b0e21ef02-podres\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.112877 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.112773    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb3c08b3-377d-4914-8c83-422b0e21ef02-sys\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.213718 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.213673    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb3c08b3-377d-4914-8c83-422b0e21ef02-proc\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.213718 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.213724    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njpd2\" (UniqueName: \"kubernetes.io/projected/bb3c08b3-377d-4914-8c83-422b0e21ef02-kube-api-access-njpd2\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.213966 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.213757    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb3c08b3-377d-4914-8c83-422b0e21ef02-lib-modules\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.213966 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.213785    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb3c08b3-377d-4914-8c83-422b0e21ef02-podres\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.213966 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.213795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb3c08b3-377d-4914-8c83-422b0e21ef02-proc\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.213966 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.213912    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb3c08b3-377d-4914-8c83-422b0e21ef02-lib-modules\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.213966 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.213919    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb3c08b3-377d-4914-8c83-422b0e21ef02-podres\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.213966 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.213910    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb3c08b3-377d-4914-8c83-422b0e21ef02-sys\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.213966 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.213960    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb3c08b3-377d-4914-8c83-422b0e21ef02-sys\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.223589 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.223562    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njpd2\" (UniqueName: \"kubernetes.io/projected/bb3c08b3-377d-4914-8c83-422b0e21ef02-kube-api-access-njpd2\") pod \"perf-node-gather-daemonset-bkts5\" (UID: \"bb3c08b3-377d-4914-8c83-422b0e21ef02\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.365405 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.365369    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:26.541343 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:26.541275    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"]
Jul 29 19:18:27.355482 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:27.355439    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5" event={"ID":"bb3c08b3-377d-4914-8c83-422b0e21ef02","Type":"ContainerStarted","Data":"48b42f713411d9914cf4b3807ed556436b2e2f6e292c4ecd02902d81235228b6"}
Jul 29 19:18:27.355984 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:27.355512    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5" event={"ID":"bb3c08b3-377d-4914-8c83-422b0e21ef02","Type":"ContainerStarted","Data":"54b288c9088b7b77697bb102e4bf6aa9a2a4f651594f5fd1a927b6adc0e74eac"}
Jul 29 19:18:27.355984 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:27.355568    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:27.373676 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:27.373621    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5" podStartSLOduration=1.3736009 podStartE2EDuration="1.3736009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:27.372081512 +0000 UTC m=+2633.266081936" watchObservedRunningTime="2026-07-29 19:18:27.3736009 +0000 UTC m=+2633.267601301"
Jul 29 19:18:27.569556 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:27.569527    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w6dzw_e3218c46-a240-4e04-b5a6-1aa3b20524ce/dns/0.log"
Jul 29 19:18:27.594109 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:27.594085    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w6dzw_e3218c46-a240-4e04-b5a6-1aa3b20524ce/kube-rbac-proxy/0.log"
Jul 29 19:18:27.676186 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:27.676108    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2qlfn_c98ed70e-1aab-43f0-b57d-192dac28efd6/dns-node-resolver/0.log"
Jul 29 19:18:28.307915 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:28.307874    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2qm5s_990500cb-5912-4252-b507-f499bae20618/node-ca/0.log"
Jul 29 19:18:29.418258 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:29.418215    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-w95c8_d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2/istio-proxy/0.log"
Jul 29 19:18:29.418975 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:29.418952    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-w95c8_d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2/istio-proxy/1.log"
Jul 29 19:18:29.999520 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:29.999484    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dw6dv_5830c3a8-6329-4806-b3f0-1cfb9141886d/serve-healthcheck-canary/0.log"
Jul 29 19:18:30.619936 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:30.619904    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bqkx8_40e67943-b279-4be0-bf71-976537ed7d5b/kube-rbac-proxy/0.log"
Jul 29 19:18:30.646472 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:30.646446    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bqkx8_40e67943-b279-4be0-bf71-976537ed7d5b/exporter/0.log"
Jul 29 19:18:30.675060 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:30.675028    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bqkx8_40e67943-b279-4be0-bf71-976537ed7d5b/extractor/0.log"
Jul 29 19:18:33.373608 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:33.373574    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-bkts5"
Jul 29 19:18:34.149310 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:34.149257    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6c559f7cd9-vrx65_a2fbd76c-54bc-4252-af02-7f32764e0cdf/manager/0.log"
Jul 29 19:18:34.216842 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:34.216810    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-78cc98c464-zhjs8_5c131702-2ddc-4aa7-a4e0-35580c76401a/manager/0.log"
Jul 29 19:18:34.243170 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:34.243143    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-h6xdl_860953da-989c-4afb-8389-7a123345af55/server/0.log"
Jul 29 19:18:34.546350 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:34.546275    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-zwkth_98de5b69-59a1-4737-8775-8c4cbb27c291/manager/0.log"
Jul 29 19:18:34.569236 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:34.569183    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-fnjkq_a1585ad1-7b3b-4006-b4c0-2b8162c20613/s3-init/0.log"
Jul 29 19:18:34.608806 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:34.608768    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-8445dbf85d-wsrf9_484a885d-83b6-4779-8e99-a6a44a0212ce/seaweedfs/0.log"
Jul 29 19:18:37.103848 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:37.103816    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-w95c8_d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2/istio-proxy/0.log"
Jul 29 19:18:37.104402 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:37.104383    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-w95c8_d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2/istio-proxy/1.log"
Jul 29 19:18:38.263273 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:38.262710    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-w95c8_d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2/istio-proxy/0.log"
Jul 29 19:18:38.263273 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:38.263184    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-w95c8_d0e400cb-9af3-4b4e-aa6f-6b34561ee5f2/istio-proxy/1.log"
Jul 29 19:18:39.922754 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:39.922727    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-spkcc_42b54cb6-14da-49df-989e-b11119cea71d/migrator/0.log"
Jul 29 19:18:39.948923 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:39.948900    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-spkcc_42b54cb6-14da-49df-989e-b11119cea71d/graceful-termination/0.log"
Jul 29 19:18:40.333668 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:40.333632    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck_a8e92808-8685-4cd5-b575-e22d16dc483a/kube-storage-version-migrator-operator/1.log"
Jul 29 19:18:40.334885 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:40.334856    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-f6qck_a8e92808-8685-4cd5-b575-e22d16dc483a/kube-storage-version-migrator-operator/0.log"
Jul 29 19:18:41.532296 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:41.532266    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hfw8j_396aaa29-be70-4f0b-bd94-a3cc455a5cd2/kube-multus-additional-cni-plugins/0.log"
Jul 29 19:18:41.557463 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:41.557432    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hfw8j_396aaa29-be70-4f0b-bd94-a3cc455a5cd2/egress-router-binary-copy/0.log"
Jul 29 19:18:41.581742 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:41.581715    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hfw8j_396aaa29-be70-4f0b-bd94-a3cc455a5cd2/cni-plugins/0.log"
Jul 29 19:18:41.606394 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:41.606370    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hfw8j_396aaa29-be70-4f0b-bd94-a3cc455a5cd2/bond-cni-plugin/0.log"
Jul 29 19:18:41.630626 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:41.630601    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hfw8j_396aaa29-be70-4f0b-bd94-a3cc455a5cd2/routeoverride-cni/0.log"
Jul 29 19:18:41.653792 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:41.653769    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hfw8j_396aaa29-be70-4f0b-bd94-a3cc455a5cd2/whereabouts-cni-bincopy/0.log"
Jul 29 19:18:41.678746 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:41.678718    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hfw8j_396aaa29-be70-4f0b-bd94-a3cc455a5cd2/whereabouts-cni/0.log"
Jul 29 19:18:41.949160 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:41.949134    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r7k7j_fb20b3c0-7818-479d-8fe3-3c5700547014/kube-multus/0.log"
Jul 29 19:18:42.005959 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:42.005931    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2jvrj_7154111a-1f26-4f41-8ba6-94617dcca1b3/network-metrics-daemon/0.log"
Jul 29 19:18:42.032694 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:42.032672    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2jvrj_7154111a-1f26-4f41-8ba6-94617dcca1b3/kube-rbac-proxy/0.log"
Jul 29 19:18:43.530183 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:43.530105    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-controller/0.log"
Jul 29 19:18:43.549540 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:43.549509    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/0.log"
Jul 29 19:18:43.549540 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:43.549529    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovn-acl-logging/1.log"
Jul 29 19:18:43.569689 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:43.569659    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/kube-rbac-proxy-node/0.log"
Jul 29 19:18:43.599733 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:43.599711    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/kube-rbac-proxy-ovn-metrics/0.log"
Jul 29 19:18:43.624604 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:43.624577    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/northd/0.log"
Jul 29 19:18:43.648498 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:43.648474    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/nbdb/0.log"
Jul 29 19:18:43.672371 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:43.672343    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/sbdb/0.log"
Jul 29 19:18:43.741118 ip-10-0-137-52 kubenswrapper[2560]: I0729 19:18:43.741079    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qlsrd_1072bb12-b457-4240-abf9-fbba51caa986/ovnkube-controller/0.log"