Jul 29 16:51:09.943062 ip-10-0-134-156 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 16:51:10.246160 ip-10-0-134-156 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 16:51:10.246160 ip-10-0-134-156 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 16:51:10.246160 ip-10-0-134-156 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 16:51:10.246160 ip-10-0-134-156 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 16:51:10.246160 ip-10-0-134-156 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 16:51:10.248010 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.247918    2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 16:51:10.252606 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252588    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 16:51:10.252606 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252606    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252611    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252614    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252618    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252621    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252625    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252629    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252633    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252637    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252640    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252643    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252646    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252649    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252652    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252655    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252658    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252661    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252663    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252666    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252668    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 16:51:10.252680 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252671    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252673    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252676    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252679    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252682    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252684    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252688    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252691    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252694    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252696    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252699    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252702    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252707    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252709    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252712    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252714    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252729    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252732    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252734    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252737    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 16:51:10.253185 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252739    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252742    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252744    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252747    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252750    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252754    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252757    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252760    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252763    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252766    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252768    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252771    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252774    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252777    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252779    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252782    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252785    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252787    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252790    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 16:51:10.253663 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252793    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252796    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252798    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252801    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252804    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252806    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252812    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252815    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252818    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252820    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252823    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252825    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252828    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252831    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252833    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252836    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252838    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252841    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252843    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252846    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 16:51:10.254183 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252849    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252851    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252854    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252856    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252859    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252862    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252864    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252867    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252870    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252873    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252875    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252878    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252881    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252884    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252886    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252889    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252892    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252894    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 16:51:10.254664 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.252897    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 16:51:10.255242 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255229    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 16:51:10.255242 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255241    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255245    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255248    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255251    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255253    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255256    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255259    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255261    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255264    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255267    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255270    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255272    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255275    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255277    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255280    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255283    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255285    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255288    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255290    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255293    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 16:51:10.255297 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255295    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255298    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255301    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255303    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255306    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255309    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255312    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255314    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255317    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255319    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255322    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255324    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255327    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255330    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255333    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255335    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255338    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255341    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255343    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 16:51:10.255814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255346    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255349    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255352    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255354    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255357    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255359    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255361    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255364    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255367    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255370    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255373    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255378    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255382    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255385    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255388    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255392    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255395    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255398    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255400    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 16:51:10.256288 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255403    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255406    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255408    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255411    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255413    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255416    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255420    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255424    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255427    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255430    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255432    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255435    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255437    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255440    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255442    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255445    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255448    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255450    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255453    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255455    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 16:51:10.256778 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255458    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255461    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255463    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255468    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255471    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255474    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255477    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255479    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255482    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255485    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255487    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255490    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255493    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255495    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255498    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255501    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255503    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255506    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255508    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255511    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 16:51:10.257263 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255922    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255927    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255930    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255932    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255935    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255938    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255941    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255943    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255946    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255948    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255951    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255954    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255957    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255959    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255962    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255965    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255969    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255972    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255975    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255978    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 16:51:10.257765 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255981    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255984    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255986    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255989    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255991    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255994    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.255997    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256000    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256003    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256005    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256008    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256013    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256017    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256020    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256023    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256025    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256028    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256031    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256033    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256036    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 16:51:10.258265 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256038    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256040    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256043    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256046    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256049    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256052    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256054    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256057    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256059    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256062    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256065    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256067    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256070    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256072    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256075    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256077    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256080    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256082    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256085    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 16:51:10.258814 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256087    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256090    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256092    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256095    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256097    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256100    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256103    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256106    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256108    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256110    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256113    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256115    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256118    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256120    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256123    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256126    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256128    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256131    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256134    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256136    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 16:51:10.259275 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256139    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256142    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256144    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256147    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256149    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256152    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256154    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256157    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256159    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256161    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256164    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256166    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256169    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256171    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256174    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256177    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256179    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256183    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256187    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256190    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 16:51:10.259762 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256266    2570 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256273    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256282    2570 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256287    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256291    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256295    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256299    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256304    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256307    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256311    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256314    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256318    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256321    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256324    2570 flags.go:64] FLAG: --cgroup-root=""
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256327    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256330    2570 flags.go:64] FLAG: --client-ca-file=""
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256333    2570 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256336    2570 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256341    2570 flags.go:64] FLAG: --cluster-domain=""
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256343    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256347    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256350    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256353    2570 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256357    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 16:51:10.260287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256360    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256363    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256366    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256369    2570 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256372    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256375    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256378    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256381    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256386    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256389    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256393    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256396    2570 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256399    2570 flags.go:64] FLAG: --enable-server="true"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256402    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256406    2570 flags.go:64] FLAG: --event-burst="100"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256409    2570 flags.go:64] FLAG: --event-qps="50"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256412    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256415    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256418    2570 flags.go:64] FLAG: --eviction-hard=""
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256422    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256425    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256428    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256430    2570 flags.go:64] FLAG: --eviction-soft=""
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256433    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256436    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 16:51:10.261004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256439    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256442    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256445    2570 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256448    2570 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256450    2570 flags.go:64] FLAG: --feature-gates=""
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256454    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256457    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256460    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256463    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256466    2570 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256469    2570 flags.go:64] FLAG: --help="false"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256472    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256475    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256477    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256480    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256484    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256488    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256491    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256494    2570 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256497    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256500    2570 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256503    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256506    2570 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256509    2570 flags.go:64] FLAG: --kube-reserved=""
Jul 29 16:51:10.261624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256512    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256514    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256517    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256520    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256523    2570 flags.go:64] FLAG: --lock-file=""
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256525    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256528    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256531    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256536    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256539    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256542    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256545    2570 flags.go:64] FLAG: --logging-format="text"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256548    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256551    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256554    2570 flags.go:64] FLAG: --manifest-url=""
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256556    2570 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256561    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256564    2570 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256568    2570 flags.go:64] FLAG: --max-pods="110"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256571    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256574    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256577    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256580    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256583    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256587    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 16:51:10.262231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256591    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256599    2570 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256601    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256605    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256608    2570 flags.go:64] FLAG: --pod-cidr=""
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256611    2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256617    2570 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256620    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256623    2570 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256626    2570 flags.go:64] FLAG: --port="10250"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256629    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256632    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e33e10abc61a86bf"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256635    2570 flags.go:64] FLAG: --qos-reserved=""
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256638    2570 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256641    2570 flags.go:64] FLAG: --register-node="true"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256644    2570 flags.go:64] FLAG: --register-with-taints=""
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256648    2570 flags.go:64] FLAG: --registry-burst="10"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256651    2570 flags.go:64] FLAG: --registry-qps="5"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256654    2570 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256657    2570 flags.go:64] FLAG: --reserved-memory=""
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256660    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256663    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256666    2570 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256669    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256672    2570 flags.go:64] FLAG: --runonce="false"
Jul 29 16:51:10.262939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256675    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256678    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256681    2570 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256684    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256687    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256690    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256693    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256696    2570 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256700    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256703    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256705    2570 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256708    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256711    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256715    2570 flags.go:64] FLAG: --system-cgroups=""
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256730    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256736    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256739    2570 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256741    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256745    2570 flags.go:64] FLAG: --tls-min-version=""
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256748    2570 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256751    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256754    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256757    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256760    2570 flags.go:64] FLAG: --v="2"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256764    2570 flags.go:64] FLAG: --version="false"
Jul 29 16:51:10.263551 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256768    2570 flags.go:64] FLAG: --vmodule=""
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256772    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.256775    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256910    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256914    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256918    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256921    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256924    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256927    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256930    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256933    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256936    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256938    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256943    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256946    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256950    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256953    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256956    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256959    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 16:51:10.264535 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256963    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256967    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256970    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256973    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256976    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256978    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256981    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256984    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256988    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256991    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256994    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256996    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.256999    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257002    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257004    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257007    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257015    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257017    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257020    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257022    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 16:51:10.265171 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257024    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257028    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257030    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257033    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257036    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257038    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257042    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257044    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257047    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257049    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257053    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257055    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257058    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257061    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257063    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257066    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257068    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257071    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257073    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257076    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 16:51:10.265672 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257078    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257081    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257083    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257086    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257089    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257091    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257094    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257096    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257102    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257104    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257107    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257109    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257112    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257115    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257117    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257120    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257123    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257125    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257129    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257131    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 16:51:10.266189 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257135    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257138    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257141    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257144    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257147    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257150    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257153    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257156    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257159    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257162    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257165    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257168    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257171    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257174    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257176    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257179    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257181    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257184    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257187    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257189    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 16:51:10.266690 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257193    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257195    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.257198    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.257203    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.265418    2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.265436    2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.265457    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.265464    2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265525    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265531    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265534    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265537    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265540    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265543    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265546    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 16:51:10.267214 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265549    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265551    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265554    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265556    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265559    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265564    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265568    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265571    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265574    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265577    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265580    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265582    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265585    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265588    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265590    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265593    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265595    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265598    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265601    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 16:51:10.267594 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265603    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265607    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265609    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265613    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265616    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265619    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265621    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265624    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265627    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265629    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265632    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265635    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265637    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265640    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265643    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265645    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265648    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265650    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265653    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 16:51:10.268176 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265656    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265660    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265663    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265666    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265669    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265672    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265675    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265678    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265682    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265685    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265687    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265690    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265692    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265695    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265698    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265701    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265704    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265707    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265710    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 16:51:10.268688 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265712    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265715    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265734    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265739    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265742    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265745    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265747    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265750    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265753    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265755    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265758    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265761    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265763    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265766    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265768    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265781    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265783    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265786    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265789    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265792    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 16:51:10.269207 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265795    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 16:51:10.269758 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265798    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 16:51:10.269758 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265801    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 16:51:10.269758 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265805    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 16:51:10.269758 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265808    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 16:51:10.269758 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265811    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 16:51:10.269758 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265814    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 16:51:10.269758 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265817    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 16:51:10.269758 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265819    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 16:51:10.269758 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265823    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 16:51:10.269758 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265825    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 16:51:10.269758 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265828    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 16:51:10.269758 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265838    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 16:51:10.269758 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265841    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 16:51:10.269758 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265844    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 16:51:10.270162 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.265850    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 16:51:10.270162 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265968    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 16:51:10.270162 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265974    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 16:51:10.270162 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265977    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 16:51:10.270162 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265980    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 16:51:10.270162 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265983    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 16:51:10.270162 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265986    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 16:51:10.270162 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265988    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 16:51:10.270162 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265991    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 16:51:10.270162 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265993    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 16:51:10.270162 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265995    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 16:51:10.270162 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.265998    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 16:51:10.270162 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266001    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 16:51:10.270162 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266004    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266006    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266009    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266011    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266014    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266017    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266019    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266022    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266024    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266027    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266029    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266032    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266035    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266037    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266040    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266043    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266045    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266048    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266051    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266053    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 16:51:10.270604 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266056    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266058    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266061    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266064    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266066    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266069    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266072    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266074    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266076    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266079    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266082    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266084    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266087    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266090    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266093    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266097    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266100    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266102    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266105    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 16:51:10.271443 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266107    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266110    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266112    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266115    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266117    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266120    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266123    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266126    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266128    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266131    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266134    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266137    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266140    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266142    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266145    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266148    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266150    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266153    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266155    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266158    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 16:51:10.272037 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266160    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266163    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266166    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266168    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266171    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266173    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266176    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266180    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266183    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266186    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266189    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266191    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266194    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266197    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266199    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266202    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266205    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266207    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266210    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 16:51:10.272906 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266212    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 16:51:10.273467 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266215    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 16:51:10.273467 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266217    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 16:51:10.273467 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266220    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 16:51:10.273467 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266222    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 16:51:10.273467 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266225    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 16:51:10.273467 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266227    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 16:51:10.273467 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266230    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 16:51:10.273467 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:10.266233    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 16:51:10.273467 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.266237    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 16:51:10.273467 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.266373    2570 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 16:51:10.273467 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.268473    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 16:51:10.273467 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.269569    2570 server.go:1019] "Starting client certificate rotation"
Jul 29 16:51:10.273467 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.269690    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 16:51:10.273467 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.269758    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 16:51:10.273847 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.271532    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 16:51:10.274066 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.274050    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 16:51:10.282642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.282623    2570 log.go:25] "Validated CRI v1 runtime API"
Jul 29 16:51:10.285259 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.285240    2570 log.go:25] "Validated CRI v1 image API"
Jul 29 16:51:10.286152 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.286136    2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 16:51:10.288412 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.288383    2570 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 97edeed2-53d2-4bb7-bd21-b6df222d49ab:/dev/nvme0n1p4 b7467b01-1e30-4cf1-959d-be0ce1b08d2a:/dev/nvme0n1p3]
Jul 29 16:51:10.288498 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.288410    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 16:51:10.293255 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.293124    2570 manager.go:216] Machine: {Timestamp:2026-07-29 16:51:10.292003926 +0000 UTC m=+0.254965068 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103248 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fbd2951629f0fc3aa52768f47f0e0 SystemUUID:ec2fbd29-5162-9f0f-c3aa-52768f47f0e0 BootID:82b0e525-49f0-42ad-a0db-fc3ff5e1a3fb Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8a:af:fc:d3:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8a:af:fc:d3:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:77:30:bf:4c:64 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 16:51:10.293255 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.293253    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 16:51:10.293367 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.293347    2570 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 16:51:10.295916 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.295881    2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 16:51:10.296065 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.295919    2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 16:51:10.296974 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.296962    2570 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 16:51:10.297009 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.296977    2570 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 16:51:10.297009 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.296986    2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 16:51:10.297522 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.297511    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 16:51:10.298278 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.298266    2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 16:51:10.300912 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.300897    2570 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 16:51:10.301050 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.301040    2570 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 16:51:10.301597 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.301587    2570 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 16:51:10.301630 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.301603    2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 16:51:10.301630 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.301617    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 16:51:10.301683 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.301631    2570 kubelet.go:398] "Adding apiserver pod source"
Jul 29 16:51:10.301683 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.301642    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 16:51:10.302080 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.302063    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 16:51:10.302141 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.302083    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 16:51:10.305647 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.305625    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 16:51:10.305808 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.305788    2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 16:51:10.307660 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.307646    2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 16:51:10.307699 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.307678    2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 16:51:10.309189 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.309173    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 16:51:10.309284 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.309195    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 16:51:10.309284 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.309204    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 16:51:10.309284 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.309212    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 16:51:10.309284 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.309225    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 16:51:10.309284 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.309234    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 16:51:10.309284 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.309246    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 16:51:10.309284 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.309254    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 16:51:10.309284 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.309264    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 16:51:10.309284 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.309272    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 16:51:10.309284 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.309285    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 16:51:10.309560 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.309294    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 16:51:10.309823 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.309811    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 16:51:10.309899 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.309825    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 16:51:10.313256 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.313238    2570 server.go:1268] "Started kubelet"
Jul 29 16:51:10.313381 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.313328    2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 16:51:10.313848 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.313408    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 16:51:10.313848 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.313492    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 16:51:10.313848 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.313824    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 16:51:10.314039 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.313868    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-156.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 16:51:10.314039 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.314015    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 16:51:10.314256 ip-10-0-134-156 systemd[1]: Started Kubernetes Kubelet.
Jul 29 16:51:10.315172 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.315104    2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 16:51:10.315359 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.315308    2570 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 16:51:10.318513 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.318491    2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 16:51:10.320061 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.319296    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-156.ec2.internal.18c6d0ca77f08f80  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-156.ec2.internal,UID:ip-10-0-134-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-156.ec2.internal,},FirstTimestamp:2026-07-29 16:51:10.313205632 +0000 UTC m=+0.276166774,LastTimestamp:2026-07-29 16:51:10.313205632 +0000 UTC m=+0.276166774,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-156.ec2.internal,}"
Jul 29 16:51:10.321704 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.321682    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 16:51:10.321820 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.321691    2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 16:51:10.321881 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.321853    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:10.322579 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.322541    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 16:51:10.322579 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.322561    2570 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 16:51:10.322579 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.322580    2570 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 16:51:10.322773 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.322578    2570 factory.go:55] Registering systemd factory
Jul 29 16:51:10.322773 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.322606    2570 factory.go:223] Registration of the systemd container factory successfully
Jul 29 16:51:10.322773 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.322698    2570 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 16:51:10.322773 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.322704    2570 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 16:51:10.322970 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.322953    2570 factory.go:153] Registering CRI-O factory
Jul 29 16:51:10.322970 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.322970    2570 factory.go:223] Registration of the crio container factory successfully
Jul 29 16:51:10.323120 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.323015    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 16:51:10.323120 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.323054    2570 factory.go:103] Registering Raw factory
Jul 29 16:51:10.323120 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.323071    2570 manager.go:1196] Started watching for new ooms in manager
Jul 29 16:51:10.323982 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.323966    2570 manager.go:349] Starting recovery of all containers
Jul 29 16:51:10.328163 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.328128    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 16:51:10.328163 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.328137    2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 16:51:10.335644 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.335619    2570 manager.go:354] Recovery completed
Jul 29 16:51:10.340272 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.340257    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:10.345380 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.345358    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:10.345479 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.345395    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:10.345479 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.345406    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:10.346010 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.345995    2570 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 16:51:10.346010 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.346008    2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 16:51:10.346116 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.346014    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:10.346116 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.346025    2570 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 16:51:10.346819 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.346796    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:10.346881 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.346830    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:10.346881 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.346841    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:10.347566 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.347479    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-156.ec2.internal.18c6d0ca79db7f4e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-156.ec2.internal,UID:ip-10-0-134-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-156.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-156.ec2.internal,},FirstTimestamp:2026-07-29 16:51:10.345379662 +0000 UTC m=+0.308340804,LastTimestamp:2026-07-29 16:51:10.345379662 +0000 UTC m=+0.308340804,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-156.ec2.internal,}"
Jul 29 16:51:10.348865 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.348847    2570 policy_none.go:49] "None policy: Start"
Jul 29 16:51:10.348865 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.348865    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 16:51:10.349004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.348876    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 16:51:10.351087 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.351076    2570 policy_none.go:47] "Start"
Jul 29 16:51:10.356358 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.356339    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-94tsl"
Jul 29 16:51:10.357218 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.357141    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-156.ec2.internal.18c6d0ca79dbd035  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-156.ec2.internal,UID:ip-10-0-134-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-156.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-156.ec2.internal,},FirstTimestamp:2026-07-29 16:51:10.345400373 +0000 UTC m=+0.308361515,LastTimestamp:2026-07-29 16:51:10.345400373 +0000 UTC m=+0.308361515,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-156.ec2.internal,}"
Jul 29 16:51:10.364431 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.364410    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-94tsl"
Jul 29 16:51:10.370292 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.370188    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-156.ec2.internal.18c6d0ca79dbf678  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-156.ec2.internal,UID:ip-10-0-134-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-156.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-156.ec2.internal,},FirstTimestamp:2026-07-29 16:51:10.345410168 +0000 UTC m=+0.308371310,LastTimestamp:2026-07-29 16:51:10.345410168 +0000 UTC m=+0.308371310,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-156.ec2.internal,}"
Jul 29 16:51:10.385662 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.385422    2570 manager.go:337] "Starting Device Plugin manager"
Jul 29 16:51:10.385850 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.385679    2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 16:51:10.385850 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.385691    2570 server.go:87] "Starting device plugin registration server"
Jul 29 16:51:10.385975 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.385962    2570 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 16:51:10.386018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.385976    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 16:51:10.386092 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.386075    2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 16:51:10.386186 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.386168    2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 16:51:10.386186 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.386182    2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 16:51:10.386755 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.386738    2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 16:51:10.386844 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.386791    2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:10.432322 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.432279    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 16:51:10.439907 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.433503    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 16:51:10.439907 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.433552    2570 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 16:51:10.439907 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.433579    2570 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 16:51:10.439907 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.433618    2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 16:51:10.439907 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.436219    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 16:51:10.486298 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.486268    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:10.487325 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.487307    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:10.487423 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.487337    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:10.487423 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.487347    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:10.487423 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.487373    2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.497345 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.497285    2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.497345 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.497313    2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-156.ec2.internal\": node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:10.528023 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.527996    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:10.534136 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.534112    2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"]
Jul 29 16:51:10.534192 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.534179    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:10.535791 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.535776    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:10.535868 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.535804    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:10.535868 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.535813    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:10.538009 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.537996    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:10.538130 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.538104    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.538166 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.538147    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:10.539926 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.539912    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:10.539988 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.539936    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:10.539988 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.539945    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:10.540059 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.539913    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:10.540059 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.540013    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:10.540059 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.540025    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:10.542164 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.542147    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.542248 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.542172    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:10.543159 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.543137    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:10.543276 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.543186    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:10.543276 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.543199    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:10.572420 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.572397    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-156.ec2.internal\" not found" node="ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.576643 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.576627    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-156.ec2.internal\" not found" node="ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.624004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.623969    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5ee567ab3aebaff2f08eb7ecd36b407-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"d5ee567ab3aebaff2f08eb7ecd36b407\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.624004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.624005    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ee567ab3aebaff2f08eb7ecd36b407-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"d5ee567ab3aebaff2f08eb7ecd36b407\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.624196 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.624031    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/159881cacdca307468786a6d29c1c03f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-156.ec2.internal\" (UID: \"159881cacdca307468786a6d29c1c03f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.629087 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.629059    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:10.725038 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.724979    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/159881cacdca307468786a6d29c1c03f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-156.ec2.internal\" (UID: \"159881cacdca307468786a6d29c1c03f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.725038 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.725048    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5ee567ab3aebaff2f08eb7ecd36b407-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"d5ee567ab3aebaff2f08eb7ecd36b407\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.725230 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.725082    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ee567ab3aebaff2f08eb7ecd36b407-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"d5ee567ab3aebaff2f08eb7ecd36b407\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.725230 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.725080    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/159881cacdca307468786a6d29c1c03f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-156.ec2.internal\" (UID: \"159881cacdca307468786a6d29c1c03f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.725230 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.725104    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5ee567ab3aebaff2f08eb7ecd36b407-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"d5ee567ab3aebaff2f08eb7ecd36b407\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.725230 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.725157    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ee567ab3aebaff2f08eb7ecd36b407-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"d5ee567ab3aebaff2f08eb7ecd36b407\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.730094 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.730074    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:10.830935 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.830898    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:10.875111 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.875082    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.880126 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:10.879818    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:10.931048 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:10.931009    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:11.031564 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:11.031541    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:11.132144 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:11.132067    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:11.232736 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:11.232674    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:11.269898 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.269857    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 16:51:11.270371 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.270031    2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 16:51:11.327296 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.327267    2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 16:51:11.333080 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:11.333055    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:11.378261 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.378220    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 16:46:11 +0000 UTC" deadline="2028-04-02 01:39:18.18177574 +0000 UTC"
Jul 29 16:51:11.378261 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.378259    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14696h48m6.803521711s"
Jul 29 16:51:11.433922 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:11.433862    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:11.438818 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.438795    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:11.440248 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.440231    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:11.440350 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.440277    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:11.440350 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.440291    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:11.534986 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:11.534940    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:11.598921 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.598892    2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 16:51:11.635368 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:11.635336    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:11.736012 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:11.735950    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Jul 29 16:51:11.755812 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.755778    2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 16:51:11.822169 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.822135    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:11.834359 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.834333    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 16:51:11.835297 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.835276    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"
Jul 29 16:51:11.841904 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.841885    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 16:51:11.934735 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:11.934694    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ee567ab3aebaff2f08eb7ecd36b407.slice/crio-6b5000d24c7067bd64a1eb2f4e453e0cc4f9552169b4e2525091da4e23fda429 WatchSource:0}: Error finding container 6b5000d24c7067bd64a1eb2f4e453e0cc4f9552169b4e2525091da4e23fda429: Status 404 returned error can't find the container with id 6b5000d24c7067bd64a1eb2f4e453e0cc4f9552169b4e2525091da4e23fda429
Jul 29 16:51:11.939951 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:11.939932    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 16:51:12.146912 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:12.146884    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 16:51:13.702544 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:13.702514    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159881cacdca307468786a6d29c1c03f.slice/crio-562485f2dbf77df31385d67723b6c973671ecf54b9c16b891f30e30df5d7acb7 WatchSource:0}: Error finding container 562485f2dbf77df31385d67723b6c973671ecf54b9c16b891f30e30df5d7acb7: Status 404 returned error can't find the container with id 562485f2dbf77df31385d67723b6c973671ecf54b9c16b891f30e30df5d7acb7
Jul 29 16:51:13.778229 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.778194    2570 apiserver.go:52] "Watching apiserver"
Jul 29 16:51:13.784433 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.784408    2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 16:51:13.784764 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.784744    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z","openshift-image-registry/node-ca-6qp9p","openshift-multus/multus-additional-cni-plugins-lsxjp","openshift-network-diagnostics/network-check-target-zb222","openshift-ovn-kubernetes/ovnkube-node-qfpbq","kube-system/global-pull-secret-syncer-dxzh5","openshift-cluster-node-tuning-operator/tuned-qzk6w","openshift-dns/node-resolver-dq655","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal","openshift-multus/multus-87796","openshift-multus/network-metrics-daemon-pmqs8","openshift-network-operator/iptables-alerter-7p58p","kube-system/konnectivity-agent-ttg74","kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"]
Jul 29 16:51:13.788095 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.788077    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.790192 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.790148    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.790292 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.790275    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 16:51:13.790346 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.790329    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lzmrr\""
Jul 29 16:51:13.790676 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.790658    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 16:51:13.790732 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.790661    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 16:51:13.792337 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.792320    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:13.792411 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:13.792389    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:13.792467 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.792408    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 16:51:13.792517 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.792495    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 16:51:13.792784 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.792768    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cgfjp\""
Jul 29 16:51:13.794482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.794462    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.795938 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.795922    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 16:51:13.796424 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.796403    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 16:51:13.796503 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.796423    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j4d56\""
Jul 29 16:51:13.797032 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.796690    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 16:51:13.797032 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.796693    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.797032 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.796841    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 16:51:13.797032 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.796867    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 16:51:13.797032 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.796929    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 16:51:13.800217 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.799406    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:13.800217 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:13.799498    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:13.800217 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.799592    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xlwpm\""
Jul 29 16:51:13.800217 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.799823    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 16:51:13.800217 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.800103    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 16:51:13.800217 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.800114    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 16:51:13.800544 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.800251    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 16:51:13.800544 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.800418    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 16:51:13.800544 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.800424    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 16:51:13.802436 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.802410    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6qp9p"
Jul 29 16:51:13.804282 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.804265    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 16:51:13.804384 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.804369    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 16:51:13.804620 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.804604    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 16:51:13.804697 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.804636    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dq655"
Jul 29 16:51:13.804804 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.804791    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6nhkp\""
Jul 29 16:51:13.806447 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.806432    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4f6z8\""
Jul 29 16:51:13.806638 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.806625    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 16:51:13.806731 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.806720    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 16:51:13.806933 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.806920    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-87796"
Jul 29 16:51:13.807239 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.807217    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 16:51:13.808136 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808118    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-cni-bin\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.808219 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808149    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-run\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.808219 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808183    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-var-lib-kubelet\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.808219 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808205    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfwns\" (UniqueName: \"kubernetes.io/projected/e356ec32-854a-4b20-985b-e93890a39aa3-kube-api-access-hfwns\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.808337 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-log-socket\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.808337 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808253    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-sys-fs\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.808337 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808276    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-sysconfig\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.808337 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808297    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-host\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.808337 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808322    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e356ec32-854a-4b20-985b-e93890a39aa3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.808564 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808360    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-kubelet\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.808564 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808385    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9243e853-a0da-492f-b653-b83b2d974747-ovnkube-config\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.808564 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808401    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9243e853-a0da-492f-b653-b83b2d974747-ovnkube-script-lib\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.808564 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808420    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxvwb\" (UniqueName: \"kubernetes.io/projected/9243e853-a0da-492f-b653-b83b2d974747-kube-api-access-zxvwb\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.808564 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808437    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-socket-dir\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.808564 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808455    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-registration-dir\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.808564 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808495    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-sys\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.808564 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808527    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e356ec32-854a-4b20-985b-e93890a39aa3-system-cni-dir\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.808564 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808561    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e356ec32-854a-4b20-985b-e93890a39aa3-os-release\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.808910 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808588    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e356ec32-854a-4b20-985b-e93890a39aa3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.808910 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808613    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-slash\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.808910 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808657    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z4bz\" (UniqueName: \"kubernetes.io/projected/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-kube-api-access-6z4bz\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.808910 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808680    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-sysctl-d\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.808910 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808701    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-lib-modules\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.808910 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808723    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-run-systemd\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.808910 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808762    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-var-lib-openvswitch\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.808910 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808781    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-run-openvswitch\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.808910 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808801    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-node-log\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.808910 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808823    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-run-ovn-kubernetes\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.808910 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808848    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.808910 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808884    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-kubernetes\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.808910 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808907    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-systemd\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808930    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-tuned\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808955    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e356ec32-854a-4b20-985b-e93890a39aa3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.808995    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-run-ovn\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809014    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d8vjg\""
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809024    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-sysctl-conf\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809040    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809042    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e356ec32-854a-4b20-985b-e93890a39aa3-cnibin\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809079    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-systemd-units\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809104    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-cni-netd\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809131    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809210    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9243e853-a0da-492f-b653-b83b2d974747-env-overrides\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809235    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9243e853-a0da-492f-b653-b83b2d974747-ovn-node-metrics-cert\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809266    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809288    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-device-dir\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809311    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-modprobe-d\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809328    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-tmp\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.809482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809352    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97qd5\" (UniqueName: \"kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5\") pod \"network-check-target-zb222\" (UID: \"335e89f7-203c-4d41-8d24-b2fde7765514\") " pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:13.809984 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809377    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e356ec32-854a-4b20-985b-e93890a39aa3-cni-binary-copy\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.809984 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809409    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-etc-openvswitch\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.809984 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809429    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:13.809984 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809436    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjbdl\" (UniqueName: \"kubernetes.io/projected/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-kube-api-access-tjbdl\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.809984 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.809458    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-run-netns\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.809984 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:13.809496    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:13.811680 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.811664    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7p58p"
Jul 29 16:51:13.813942 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.813921    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 16:51:13.814192 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.813942    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 16:51:13.814192 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.813957    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 16:51:13.814192 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.813956    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tjzmz\""
Jul 29 16:51:13.814192 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.813957    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ttg74"
Jul 29 16:51:13.816054 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.816040    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 16:51:13.816255 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.816128    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mq7gs\""
Jul 29 16:51:13.816353 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.816339    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 16:51:13.829420 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.829403    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b7mfm"
Jul 29 16:51:13.836136 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.836119    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b7mfm"
Jul 29 16:51:13.897050 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.896969    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 16:51:13.909803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.909767    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e356ec32-854a-4b20-985b-e93890a39aa3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.909972 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.909813    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-run-ovn\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.909972 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.909843    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d4a05de-792b-4584-8f29-8f396f5f59fb-host-slash\") pod \"iptables-alerter-7p58p\" (UID: \"4d4a05de-792b-4584-8f29-8f396f5f59fb\") " pod="openshift-network-operator/iptables-alerter-7p58p"
Jul 29 16:51:13.909972 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.909868    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpp9w\" (UniqueName: \"kubernetes.io/projected/4d4a05de-792b-4584-8f29-8f396f5f59fb-kube-api-access-mpp9w\") pod \"iptables-alerter-7p58p\" (UID: \"4d4a05de-792b-4584-8f29-8f396f5f59fb\") " pod="openshift-network-operator/iptables-alerter-7p58p"
Jul 29 16:51:13.909972 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.909894    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-system-cni-dir\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.909972 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.909938    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-run-ovn\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.909972 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.909947    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e356ec32-854a-4b20-985b-e93890a39aa3-cnibin\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.910254 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.909989    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-systemd-units\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.910254 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.909988    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e356ec32-854a-4b20-985b-e93890a39aa3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.910254 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910009    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e356ec32-854a-4b20-985b-e93890a39aa3-cnibin\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.910254 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910046    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-cni-netd\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.910254 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910067    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9243e853-a0da-492f-b653-b83b2d974747-env-overrides\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.910254 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910102    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f28253eb-90bd-48ad-ade7-a48e9fab2605-agent-certs\") pod \"konnectivity-agent-ttg74\" (UID: \"f28253eb-90bd-48ad-ade7-a48e9fab2605\") " pod="kube-system/konnectivity-agent-ttg74"
Jul 29 16:51:13.910254 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910105    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-cni-netd\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.910254 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910149    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f28253eb-90bd-48ad-ade7-a48e9fab2605-konnectivity-ca\") pod \"konnectivity-agent-ttg74\" (UID: \"f28253eb-90bd-48ad-ade7-a48e9fab2605\") " pod="kube-system/konnectivity-agent-ttg74"
Jul 29 16:51:13.910254 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910188    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-systemd-units\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.910642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910289    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-os-release\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.910642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910314    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-run-netns\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.910642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910343    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.910642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910377    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-device-dir\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.910642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910392    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.910642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910413    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-tmp\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.910642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910462    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-run-k8s-cni-cncf-io\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.910642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910471    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-device-dir\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.910642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910514    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-var-lib-cni-bin\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.910642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910551    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79ppj\" (UniqueName: \"kubernetes.io/projected/438c3501-4812-489f-ae51-a25e01b9aacc-kube-api-access-79ppj\") pod \"network-metrics-daemon-pmqs8\" (UID: \"438c3501-4812-489f-ae51-a25e01b9aacc\") " pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:13.910642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910588    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjbdl\" (UniqueName: \"kubernetes.io/projected/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-kube-api-access-tjbdl\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.910642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910615    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-run-netns\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.910642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910641    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77wml\" (UniqueName: \"kubernetes.io/projected/e22624ca-991d-4e4d-8db7-02119caf746d-kube-api-access-77wml\") pod \"node-ca-6qp9p\" (UID: \"e22624ca-991d-4e4d-8db7-02119caf746d\") " pod="openshift-image-registry/node-ca-6qp9p"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910666    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs\") pod \"network-metrics-daemon-pmqs8\" (UID: \"438c3501-4812-489f-ae51-a25e01b9aacc\") " pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910671    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-run-netns\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910675    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9243e853-a0da-492f-b653-b83b2d974747-env-overrides\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910692    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-var-lib-kubelet\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910739    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-var-lib-kubelet\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910746    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-log-socket\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910776    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22624ca-991d-4e4d-8db7-02119caf746d-host\") pod \"node-ca-6qp9p\" (UID: \"e22624ca-991d-4e4d-8db7-02119caf746d\") " pod="openshift-image-registry/node-ca-6qp9p"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910801    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-multus-daemon-config\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910812    2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910821    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-log-socket\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910843    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-sys-fs\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910868    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-host\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910890    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e356ec32-854a-4b20-985b-e93890a39aa3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910910    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-kubelet\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910935    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9243e853-a0da-492f-b653-b83b2d974747-ovnkube-script-lib\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910941    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-sys-fs\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910958    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxvwb\" (UniqueName: \"kubernetes.io/projected/9243e853-a0da-492f-b653-b83b2d974747-kube-api-access-zxvwb\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.911237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910971    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-host\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.910976    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-socket-dir\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911007    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-kubelet\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911028    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e356ec32-854a-4b20-985b-e93890a39aa3-system-cni-dir\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911075    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e356ec32-854a-4b20-985b-e93890a39aa3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911126    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e356ec32-854a-4b20-985b-e93890a39aa3-system-cni-dir\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911156    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-var-lib-openvswitch\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911195    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9243e853-a0da-492f-b653-b83b2d974747-ovn-node-metrics-cert\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-run-multus-certs\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911252    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-lib-modules\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911279    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1bd992f-213e-46bd-87a6-b06f7895719d-dbus\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911299    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjw2v\" (UniqueName: \"kubernetes.io/projected/8ed06670-f24d-4521-815a-0d27741edac3-kube-api-access-kjw2v\") pod \"node-resolver-dq655\" (UID: \"8ed06670-f24d-4521-815a-0d27741edac3\") " pod="openshift-dns/node-resolver-dq655"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911448    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-lib-modules\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911555    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e356ec32-854a-4b20-985b-e93890a39aa3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911544    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9243e853-a0da-492f-b653-b83b2d974747-ovnkube-script-lib\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911572    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" event={"ID":"159881cacdca307468786a6d29c1c03f","Type":"ContainerStarted","Data":"562485f2dbf77df31385d67723b6c973671ecf54b9c16b891f30e30df5d7acb7"}
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911641    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-socket-dir\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.912000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911648    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e356ec32-854a-4b20-985b-e93890a39aa3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911650    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-tuned\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911701    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-cni-binary-copy\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911717    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-var-lib-kubelet\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911761    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-var-lib-openvswitch\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911760    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-multus-conf-dir\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911796    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-sysctl-conf\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911818    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911844    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ed06670-f24d-4521-815a-0d27741edac3-hosts-file\") pod \"node-resolver-dq655\" (UID: \"8ed06670-f24d-4521-815a-0d27741edac3\") " pod="openshift-dns/node-resolver-dq655"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911868    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-cnibin\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911891    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw46b\" (UniqueName: \"kubernetes.io/projected/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-kube-api-access-cw46b\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911929    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-modprobe-d\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911952    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qd5\" (UniqueName: \"kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5\") pod \"network-check-target-zb222\" (UID: \"335e89f7-203c-4d41-8d24-b2fde7765514\") " pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911956    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-sysctl-conf\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.911978    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e356ec32-854a-4b20-985b-e93890a39aa3-cni-binary-copy\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912003    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-etc-openvswitch\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912029    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-multus-cni-dir\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.912803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912053    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-var-lib-cni-multus\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912004    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912054    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-modprobe-d\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912087    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-cni-bin\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912114    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-run\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912139    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwns\" (UniqueName: \"kubernetes.io/projected/e356ec32-854a-4b20-985b-e93890a39aa3-kube-api-access-hfwns\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912246    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-cni-bin\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912301    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-etc-openvswitch\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912410    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-run\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912550    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e356ec32-854a-4b20-985b-e93890a39aa3-cni-binary-copy\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912778    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" event={"ID":"d5ee567ab3aebaff2f08eb7ecd36b407","Type":"ContainerStarted","Data":"6b5000d24c7067bd64a1eb2f4e453e0cc4f9552169b4e2525091da4e23fda429"}
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912792    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-multus-socket-dir-parent\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912837    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-hostroot\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912863    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-etc-kubernetes\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912888    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-sysconfig\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912927    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9243e853-a0da-492f-b653-b83b2d974747-ovnkube-config\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912945    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-sysconfig\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.913561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.912960    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-registration-dir\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913002    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-sys\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913034    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e356ec32-854a-4b20-985b-e93890a39aa3-os-release\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913085    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-registration-dir\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913101    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e356ec32-854a-4b20-985b-e93890a39aa3-os-release\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913087    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-sys\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913111    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-slash\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913141    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-run-systemd\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913149    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-slash\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913181    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-run-openvswitch\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913206    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-node-log\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913231    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z4bz\" (UniqueName: \"kubernetes.io/projected/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-kube-api-access-6z4bz\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913235    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-run-openvswitch\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913206    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-run-systemd\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913254    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-sysctl-d\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913278    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-node-log\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913279    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-run-ovn-kubernetes\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.914112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913305    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1bd992f-213e-46bd-87a6-b06f7895719d-kubelet-config\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913313    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9243e853-a0da-492f-b653-b83b2d974747-host-run-ovn-kubernetes\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913326    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913419    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-sysctl-d\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913438    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9243e853-a0da-492f-b653-b83b2d974747-ovnkube-config\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913483    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e22624ca-991d-4e4d-8db7-02119caf746d-serviceca\") pod \"node-ca-6qp9p\" (UID: \"e22624ca-991d-4e4d-8db7-02119caf746d\") " pod="openshift-image-registry/node-ca-6qp9p"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913507    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed06670-f24d-4521-815a-0d27741edac3-tmp-dir\") pod \"node-resolver-dq655\" (UID: \"8ed06670-f24d-4521-815a-0d27741edac3\") " pod="openshift-dns/node-resolver-dq655"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913529    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4d4a05de-792b-4584-8f29-8f396f5f59fb-iptables-alerter-script\") pod \"iptables-alerter-7p58p\" (UID: \"4d4a05de-792b-4584-8f29-8f396f5f59fb\") " pod="openshift-network-operator/iptables-alerter-7p58p"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913557    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913584    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-kubernetes\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913610    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-systemd\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913669    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-kubernetes\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913669    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.913711    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-systemd\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.914337    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-etc-tuned\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.914379    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-tmp\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.914641 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.914448    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9243e853-a0da-492f-b653-b83b2d974747-ovn-node-metrics-cert\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.918431 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:13.918408    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 16:51:13.918540 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:13.918436    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 16:51:13.918540 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:13.918449    2570 projected.go:196] Error preparing data for projected volume kube-api-access-97qd5 for pod openshift-network-diagnostics/network-check-target-zb222: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:13.918540 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.918523    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxvwb\" (UniqueName: \"kubernetes.io/projected/9243e853-a0da-492f-b653-b83b2d974747-kube-api-access-zxvwb\") pod \"ovnkube-node-qfpbq\" (UID: \"9243e853-a0da-492f-b653-b83b2d974747\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:13.918540 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:13.918537    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5 podName:335e89f7-203c-4d41-8d24-b2fde7765514 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:14.418504143 +0000 UTC m=+2.908023796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-97qd5" (UniqueName: "kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5") pod "network-check-target-zb222" (UID: "335e89f7-203c-4d41-8d24-b2fde7765514") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:13.918735 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.918628    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjbdl\" (UniqueName: \"kubernetes.io/projected/ea15cf35-0332-4188-8b22-c4f58a5b9a9f-kube-api-access-tjbdl\") pod \"tuned-qzk6w\" (UID: \"ea15cf35-0332-4188-8b22-c4f58a5b9a9f\") " pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:13.919539 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.919524    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfwns\" (UniqueName: \"kubernetes.io/projected/e356ec32-854a-4b20-985b-e93890a39aa3-kube-api-access-hfwns\") pod \"multus-additional-cni-plugins-lsxjp\" (UID: \"e356ec32-854a-4b20-985b-e93890a39aa3\") " pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:13.923259 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:13.923241    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z4bz\" (UniqueName: \"kubernetes.io/projected/f7b01bca-6c8a-48e4-be26-e754c4d9df3c-kube-api-access-6z4bz\") pod \"aws-ebs-csi-driver-node-jxr4z\" (UID: \"f7b01bca-6c8a-48e4-be26-e754c4d9df3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:14.014523 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014494    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-run-multus-certs\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.014523 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014525    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1bd992f-213e-46bd-87a6-b06f7895719d-dbus\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:14.014778 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014546    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjw2v\" (UniqueName: \"kubernetes.io/projected/8ed06670-f24d-4521-815a-0d27741edac3-kube-api-access-kjw2v\") pod \"node-resolver-dq655\" (UID: \"8ed06670-f24d-4521-815a-0d27741edac3\") " pod="openshift-dns/node-resolver-dq655"
Jul 29 16:51:14.014778 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014572    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-cni-binary-copy\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.014778 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014589    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-var-lib-kubelet\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.014778 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014604    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-multus-conf-dir\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.014778 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014621    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-run-multus-certs\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.014778 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014678    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-var-lib-kubelet\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.014778 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014679    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-multus-conf-dir\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.014778 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014627    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ed06670-f24d-4521-815a-0d27741edac3-hosts-file\") pod \"node-resolver-dq655\" (UID: \"8ed06670-f24d-4521-815a-0d27741edac3\") " pod="openshift-dns/node-resolver-dq655"
Jul 29 16:51:14.014778 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014721    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-cnibin\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.014778 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014748    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw46b\" (UniqueName: \"kubernetes.io/projected/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-kube-api-access-cw46b\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014824    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-multus-cni-dir\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014851    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-var-lib-cni-multus\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014881    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-multus-socket-dir-parent\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014905    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-hostroot\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014940    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1bd992f-213e-46bd-87a6-b06f7895719d-dbus\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014945    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-etc-kubernetes\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014979    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-etc-kubernetes\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.014989    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-multus-cni-dir\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015015    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1bd992f-213e-46bd-87a6-b06f7895719d-kubelet-config\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015025    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-hostroot\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015034    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-multus-socket-dir-parent\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015042    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015057    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-var-lib-cni-multus\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015083    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-cnibin\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015067    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e22624ca-991d-4e4d-8db7-02119caf746d-serviceca\") pod \"node-ca-6qp9p\" (UID: \"e22624ca-991d-4e4d-8db7-02119caf746d\") " pod="openshift-image-registry/node-ca-6qp9p"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015131    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed06670-f24d-4521-815a-0d27741edac3-tmp-dir\") pod \"node-resolver-dq655\" (UID: \"8ed06670-f24d-4521-815a-0d27741edac3\") " pod="openshift-dns/node-resolver-dq655"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015134    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1bd992f-213e-46bd-87a6-b06f7895719d-kubelet-config\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:14.015293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015156    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4d4a05de-792b-4584-8f29-8f396f5f59fb-iptables-alerter-script\") pod \"iptables-alerter-7p58p\" (UID: \"4d4a05de-792b-4584-8f29-8f396f5f59fb\") " pod="openshift-network-operator/iptables-alerter-7p58p"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015262    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-cni-binary-copy\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015351    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ed06670-f24d-4521-815a-0d27741edac3-hosts-file\") pod \"node-resolver-dq655\" (UID: \"8ed06670-f24d-4521-815a-0d27741edac3\") " pod="openshift-dns/node-resolver-dq655"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:14.015440    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015465    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed06670-f24d-4521-815a-0d27741edac3-tmp-dir\") pod \"node-resolver-dq655\" (UID: \"8ed06670-f24d-4521-815a-0d27741edac3\") " pod="openshift-dns/node-resolver-dq655"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:14.015514    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret podName:b1bd992f-213e-46bd-87a6-b06f7895719d nodeName:}" failed. No retries permitted until 2026-07-29 16:51:14.515492396 +0000 UTC m=+3.005012033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret") pod "global-pull-secret-syncer-dxzh5" (UID: "b1bd992f-213e-46bd-87a6-b06f7895719d") : object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015600    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e22624ca-991d-4e4d-8db7-02119caf746d-serviceca\") pod \"node-ca-6qp9p\" (UID: \"e22624ca-991d-4e4d-8db7-02119caf746d\") " pod="openshift-image-registry/node-ca-6qp9p"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015662    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d4a05de-792b-4584-8f29-8f396f5f59fb-host-slash\") pod \"iptables-alerter-7p58p\" (UID: \"4d4a05de-792b-4584-8f29-8f396f5f59fb\") " pod="openshift-network-operator/iptables-alerter-7p58p"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015690    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpp9w\" (UniqueName: \"kubernetes.io/projected/4d4a05de-792b-4584-8f29-8f396f5f59fb-kube-api-access-mpp9w\") pod \"iptables-alerter-7p58p\" (UID: \"4d4a05de-792b-4584-8f29-8f396f5f59fb\") " pod="openshift-network-operator/iptables-alerter-7p58p"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015715    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-system-cni-dir\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015717    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d4a05de-792b-4584-8f29-8f396f5f59fb-host-slash\") pod \"iptables-alerter-7p58p\" (UID: \"4d4a05de-792b-4584-8f29-8f396f5f59fb\") " pod="openshift-network-operator/iptables-alerter-7p58p"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015743    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f28253eb-90bd-48ad-ade7-a48e9fab2605-agent-certs\") pod \"konnectivity-agent-ttg74\" (UID: \"f28253eb-90bd-48ad-ade7-a48e9fab2605\") " pod="kube-system/konnectivity-agent-ttg74"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015771    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f28253eb-90bd-48ad-ade7-a48e9fab2605-konnectivity-ca\") pod \"konnectivity-agent-ttg74\" (UID: \"f28253eb-90bd-48ad-ade7-a48e9fab2605\") " pod="kube-system/konnectivity-agent-ttg74"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015782    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-system-cni-dir\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015795    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-os-release\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015817    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-run-netns\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015855    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-run-k8s-cni-cncf-io\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015879    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-var-lib-cni-bin\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.016183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015902    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79ppj\" (UniqueName: \"kubernetes.io/projected/438c3501-4812-489f-ae51-a25e01b9aacc-kube-api-access-79ppj\") pod \"network-metrics-daemon-pmqs8\" (UID: \"438c3501-4812-489f-ae51-a25e01b9aacc\") " pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:14.016944 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015927    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77wml\" (UniqueName: \"kubernetes.io/projected/e22624ca-991d-4e4d-8db7-02119caf746d-kube-api-access-77wml\") pod \"node-ca-6qp9p\" (UID: \"e22624ca-991d-4e4d-8db7-02119caf746d\") " pod="openshift-image-registry/node-ca-6qp9p"
Jul 29 16:51:14.016944 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015960    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs\") pod \"network-metrics-daemon-pmqs8\" (UID: \"438c3501-4812-489f-ae51-a25e01b9aacc\") " pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:14.016944 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015969    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-var-lib-cni-bin\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.016944 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015983    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22624ca-991d-4e4d-8db7-02119caf746d-host\") pod \"node-ca-6qp9p\" (UID: \"e22624ca-991d-4e4d-8db7-02119caf746d\") " pod="openshift-image-registry/node-ca-6qp9p"
Jul 29 16:51:14.016944 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015983    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-run-k8s-cni-cncf-io\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.016944 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.015981    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-host-run-netns\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.016944 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.016020    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-multus-daemon-config\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.016944 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.016045    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-os-release\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.016944 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:14.016137    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:14.016944 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:14.016212    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs podName:438c3501-4812-489f-ae51-a25e01b9aacc nodeName:}" failed. No retries permitted until 2026-07-29 16:51:14.516184144 +0000 UTC m=+3.005703793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs") pod "network-metrics-daemon-pmqs8" (UID: "438c3501-4812-489f-ae51-a25e01b9aacc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:14.016944 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.016242    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22624ca-991d-4e4d-8db7-02119caf746d-host\") pod \"node-ca-6qp9p\" (UID: \"e22624ca-991d-4e4d-8db7-02119caf746d\") " pod="openshift-image-registry/node-ca-6qp9p"
Jul 29 16:51:14.016944 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.016522    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4d4a05de-792b-4584-8f29-8f396f5f59fb-iptables-alerter-script\") pod \"iptables-alerter-7p58p\" (UID: \"4d4a05de-792b-4584-8f29-8f396f5f59fb\") " pod="openshift-network-operator/iptables-alerter-7p58p"
Jul 29 16:51:14.016944 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.016550    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-multus-daemon-config\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.017401 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.017270    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f28253eb-90bd-48ad-ade7-a48e9fab2605-konnectivity-ca\") pod \"konnectivity-agent-ttg74\" (UID: \"f28253eb-90bd-48ad-ade7-a48e9fab2605\") " pod="kube-system/konnectivity-agent-ttg74"
Jul 29 16:51:14.018476 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.018454    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f28253eb-90bd-48ad-ade7-a48e9fab2605-agent-certs\") pod \"konnectivity-agent-ttg74\" (UID: \"f28253eb-90bd-48ad-ade7-a48e9fab2605\") " pod="kube-system/konnectivity-agent-ttg74"
Jul 29 16:51:14.025211 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.025175    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw46b\" (UniqueName: \"kubernetes.io/projected/0e13b680-aa89-4a92-8ff2-1eb3e0a0309b-kube-api-access-cw46b\") pod \"multus-87796\" (UID: \"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b\") " pod="openshift-multus/multus-87796"
Jul 29 16:51:14.025656 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.025636    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpp9w\" (UniqueName: \"kubernetes.io/projected/4d4a05de-792b-4584-8f29-8f396f5f59fb-kube-api-access-mpp9w\") pod \"iptables-alerter-7p58p\" (UID: \"4d4a05de-792b-4584-8f29-8f396f5f59fb\") " pod="openshift-network-operator/iptables-alerter-7p58p"
Jul 29 16:51:14.025972 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.025951    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77wml\" (UniqueName: \"kubernetes.io/projected/e22624ca-991d-4e4d-8db7-02119caf746d-kube-api-access-77wml\") pod \"node-ca-6qp9p\" (UID: \"e22624ca-991d-4e4d-8db7-02119caf746d\") " pod="openshift-image-registry/node-ca-6qp9p"
Jul 29 16:51:14.026094 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.026073    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjw2v\" (UniqueName: \"kubernetes.io/projected/8ed06670-f24d-4521-815a-0d27741edac3-kube-api-access-kjw2v\") pod \"node-resolver-dq655\" (UID: \"8ed06670-f24d-4521-815a-0d27741edac3\") " pod="openshift-dns/node-resolver-dq655"
Jul 29 16:51:14.027506 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.027482    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79ppj\" (UniqueName: \"kubernetes.io/projected/438c3501-4812-489f-ae51-a25e01b9aacc-kube-api-access-79ppj\") pod \"network-metrics-daemon-pmqs8\" (UID: \"438c3501-4812-489f-ae51-a25e01b9aacc\") " pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:14.097989 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.097950    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z"
Jul 29 16:51:14.107411 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.107385    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qzk6w"
Jul 29 16:51:14.108910 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:14.108880    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b01bca_6c8a_48e4_be26_e754c4d9df3c.slice/crio-3fbfd6eb16e70e5fb43fd9efcaf8710a890c298d3c85d6eccae2b77540cca468 WatchSource:0}: Error finding container 3fbfd6eb16e70e5fb43fd9efcaf8710a890c298d3c85d6eccae2b77540cca468: Status 404 returned error can't find the container with id 3fbfd6eb16e70e5fb43fd9efcaf8710a890c298d3c85d6eccae2b77540cca468
Jul 29 16:51:14.115093 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.115068    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lsxjp"
Jul 29 16:51:14.116528 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:14.116506    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea15cf35_0332_4188_8b22_c4f58a5b9a9f.slice/crio-accb270fb981f07b214d533dd2deef1e4f2cc6c889f0af7bd297fbede67ca936 WatchSource:0}: Error finding container accb270fb981f07b214d533dd2deef1e4f2cc6c889f0af7bd297fbede67ca936: Status 404 returned error can't find the container with id accb270fb981f07b214d533dd2deef1e4f2cc6c889f0af7bd297fbede67ca936
Jul 29 16:51:14.119416 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.119391    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:14.125074 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.125050    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6qp9p"
Jul 29 16:51:14.126334 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:14.126310    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode356ec32_854a_4b20_985b_e93890a39aa3.slice/crio-0a5ea3c6479f4af2ffd0dd62c21de623f27999f036ac805ffa87786f1e23eca4 WatchSource:0}: Error finding container 0a5ea3c6479f4af2ffd0dd62c21de623f27999f036ac805ffa87786f1e23eca4: Status 404 returned error can't find the container with id 0a5ea3c6479f4af2ffd0dd62c21de623f27999f036ac805ffa87786f1e23eca4
Jul 29 16:51:14.129471 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:14.129449    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9243e853_a0da_492f_b653_b83b2d974747.slice/crio-ddb41bec0c7198f8b818af1ba7963dfb5d54583184caf21a898f449c6c59090e WatchSource:0}: Error finding container ddb41bec0c7198f8b818af1ba7963dfb5d54583184caf21a898f449c6c59090e: Status 404 returned error can't find the container with id ddb41bec0c7198f8b818af1ba7963dfb5d54583184caf21a898f449c6c59090e
Jul 29 16:51:14.130293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.130252    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dq655"
Jul 29 16:51:14.135498 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.135480    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-87796"
Jul 29 16:51:14.135753 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:14.135729    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22624ca_991d_4e4d_8db7_02119caf746d.slice/crio-30a204a1ea16fe8963c404e9292b6e9dced6ee4566231c0793a94ce5c2b9756d WatchSource:0}: Error finding container 30a204a1ea16fe8963c404e9292b6e9dced6ee4566231c0793a94ce5c2b9756d: Status 404 returned error can't find the container with id 30a204a1ea16fe8963c404e9292b6e9dced6ee4566231c0793a94ce5c2b9756d
Jul 29 16:51:14.140195 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:14.140157    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed06670_f24d_4521_815a_0d27741edac3.slice/crio-aba65dc17577b492753e5e29a223395dfd79cb433c7e5c88e48c217c1f8afb60 WatchSource:0}: Error finding container aba65dc17577b492753e5e29a223395dfd79cb433c7e5c88e48c217c1f8afb60: Status 404 returned error can't find the container with id aba65dc17577b492753e5e29a223395dfd79cb433c7e5c88e48c217c1f8afb60
Jul 29 16:51:14.141240 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.141226    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7p58p"
Jul 29 16:51:14.145767 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.145747    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ttg74"
Jul 29 16:51:14.145916 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:14.145895    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e13b680_aa89_4a92_8ff2_1eb3e0a0309b.slice/crio-740b91a773b6579f32b7430028f4ee3e481061331f87d3127ce7f1cd4182f56f WatchSource:0}: Error finding container 740b91a773b6579f32b7430028f4ee3e481061331f87d3127ce7f1cd4182f56f: Status 404 returned error can't find the container with id 740b91a773b6579f32b7430028f4ee3e481061331f87d3127ce7f1cd4182f56f
Jul 29 16:51:14.151242 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:14.151219    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4a05de_792b_4584_8f29_8f396f5f59fb.slice/crio-9394c26499aadc617b0da406debcbde7c5c11cf6932684babea6b075a1085054 WatchSource:0}: Error finding container 9394c26499aadc617b0da406debcbde7c5c11cf6932684babea6b075a1085054: Status 404 returned error can't find the container with id 9394c26499aadc617b0da406debcbde7c5c11cf6932684babea6b075a1085054
Jul 29 16:51:14.155789 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:14.155768    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28253eb_90bd_48ad_ade7_a48e9fab2605.slice/crio-af965b1d3977efb5e5fd2061da9f066edee092909126605eda528cff64440560 WatchSource:0}: Error finding container af965b1d3977efb5e5fd2061da9f066edee092909126605eda528cff64440560: Status 404 returned error can't find the container with id af965b1d3977efb5e5fd2061da9f066edee092909126605eda528cff64440560
Jul 29 16:51:14.518942 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.518902    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qd5\" (UniqueName: \"kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5\") pod \"network-check-target-zb222\" (UID: \"335e89f7-203c-4d41-8d24-b2fde7765514\") " pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:14.519123 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.518969    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:14.519123 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.519001    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs\") pod \"network-metrics-daemon-pmqs8\" (UID: \"438c3501-4812-489f-ae51-a25e01b9aacc\") " pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:14.519123 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:14.519055    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 16:51:14.519123 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:14.519084    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 16:51:14.519123 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:14.519100    2570 projected.go:196] Error preparing data for projected volume kube-api-access-97qd5 for pod openshift-network-diagnostics/network-check-target-zb222: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:14.519123 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:14.519116    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:14.519459 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:14.519151    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5 podName:335e89f7-203c-4d41-8d24-b2fde7765514 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:15.519135561 +0000 UTC m=+4.008655195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-97qd5" (UniqueName: "kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5") pod "network-check-target-zb222" (UID: "335e89f7-203c-4d41-8d24-b2fde7765514") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:14.519459 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:14.519116    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:14.519459 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:14.519179    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs podName:438c3501-4812-489f-ae51-a25e01b9aacc nodeName:}" failed. No retries permitted until 2026-07-29 16:51:15.519172042 +0000 UTC m=+4.008691675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs") pod "network-metrics-daemon-pmqs8" (UID: "438c3501-4812-489f-ae51-a25e01b9aacc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:14.519459 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:14.519207    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret podName:b1bd992f-213e-46bd-87a6-b06f7895719d nodeName:}" failed. No retries permitted until 2026-07-29 16:51:15.519189711 +0000 UTC m=+4.008709362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret") pod "global-pull-secret-syncer-dxzh5" (UID: "b1bd992f-213e-46bd-87a6-b06f7895719d") : object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:14.839405 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.839303    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 16:46:13 +0000 UTC" deadline="2028-02-25 03:37:51.893873476 +0000 UTC"
Jul 29 16:51:14.839405 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.839363    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13810h46m37.054514604s"
Jul 29 16:51:14.908266 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.908229    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:14.908444 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:14.908372    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:14.920568 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.920533    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dq655" event={"ID":"8ed06670-f24d-4521-815a-0d27741edac3","Type":"ContainerStarted","Data":"aba65dc17577b492753e5e29a223395dfd79cb433c7e5c88e48c217c1f8afb60"}
Jul 29 16:51:14.924270 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.922144    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6qp9p" event={"ID":"e22624ca-991d-4e4d-8db7-02119caf746d","Type":"ContainerStarted","Data":"30a204a1ea16fe8963c404e9292b6e9dced6ee4566231c0793a94ce5c2b9756d"}
Jul 29 16:51:14.927792 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.927736    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq" event={"ID":"9243e853-a0da-492f-b653-b83b2d974747","Type":"ContainerStarted","Data":"ddb41bec0c7198f8b818af1ba7963dfb5d54583184caf21a898f449c6c59090e"}
Jul 29 16:51:14.934989 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.934941    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qzk6w" event={"ID":"ea15cf35-0332-4188-8b22-c4f58a5b9a9f","Type":"ContainerStarted","Data":"accb270fb981f07b214d533dd2deef1e4f2cc6c889f0af7bd297fbede67ca936"}
Jul 29 16:51:14.940331 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.940247    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7p58p" event={"ID":"4d4a05de-792b-4584-8f29-8f396f5f59fb","Type":"ContainerStarted","Data":"9394c26499aadc617b0da406debcbde7c5c11cf6932684babea6b075a1085054"}
Jul 29 16:51:14.943153 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.943072    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-87796" event={"ID":"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b","Type":"ContainerStarted","Data":"740b91a773b6579f32b7430028f4ee3e481061331f87d3127ce7f1cd4182f56f"}
Jul 29 16:51:14.953302 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.953267    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxjp" event={"ID":"e356ec32-854a-4b20-985b-e93890a39aa3","Type":"ContainerStarted","Data":"0a5ea3c6479f4af2ffd0dd62c21de623f27999f036ac805ffa87786f1e23eca4"}
Jul 29 16:51:14.958078 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.958046    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z" event={"ID":"f7b01bca-6c8a-48e4-be26-e754c4d9df3c","Type":"ContainerStarted","Data":"3fbfd6eb16e70e5fb43fd9efcaf8710a890c298d3c85d6eccae2b77540cca468"}
Jul 29 16:51:14.964349 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:14.963609    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ttg74" event={"ID":"f28253eb-90bd-48ad-ade7-a48e9fab2605","Type":"ContainerStarted","Data":"af965b1d3977efb5e5fd2061da9f066edee092909126605eda528cff64440560"}
Jul 29 16:51:15.530264 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:15.529633    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:15.530264 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:15.529943    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs\") pod \"network-metrics-daemon-pmqs8\" (UID: \"438c3501-4812-489f-ae51-a25e01b9aacc\") " pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:15.530264 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:15.529997    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qd5\" (UniqueName: \"kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5\") pod \"network-check-target-zb222\" (UID: \"335e89f7-203c-4d41-8d24-b2fde7765514\") " pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:15.530264 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:15.529799    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:15.530264 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:15.530104    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret podName:b1bd992f-213e-46bd-87a6-b06f7895719d nodeName:}" failed. No retries permitted until 2026-07-29 16:51:17.530081445 +0000 UTC m=+6.019601082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret") pod "global-pull-secret-syncer-dxzh5" (UID: "b1bd992f-213e-46bd-87a6-b06f7895719d") : object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:15.530264 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:15.530112    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:15.530264 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:15.530141    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 16:51:15.530264 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:15.530178    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 16:51:15.530264 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:15.530194    2570 projected.go:196] Error preparing data for projected volume kube-api-access-97qd5 for pod openshift-network-diagnostics/network-check-target-zb222: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:15.530264 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:15.530149    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs podName:438c3501-4812-489f-ae51-a25e01b9aacc nodeName:}" failed. No retries permitted until 2026-07-29 16:51:17.530138362 +0000 UTC m=+6.019658009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs") pod "network-metrics-daemon-pmqs8" (UID: "438c3501-4812-489f-ae51-a25e01b9aacc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:15.530264 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:15.530264    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5 podName:335e89f7-203c-4d41-8d24-b2fde7765514 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:17.53025173 +0000 UTC m=+6.019771363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-97qd5" (UniqueName: "kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5") pod "network-check-target-zb222" (UID: "335e89f7-203c-4d41-8d24-b2fde7765514") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:15.840366 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:15.840314    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 16:46:13 +0000 UTC" deadline="2027-12-23 13:25:42.361282608 +0000 UTC"
Jul 29 16:51:15.840366 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:15.840363    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12284h34m26.520924559s"
Jul 29 16:51:15.907877 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:15.907371    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:15.907877 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:15.907513    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:15.908360 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:15.908217    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:15.908360 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:15.908323    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:15.998651 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:15.998612    2570 generic.go:356] "Generic (PLEG): container finished" podID="d5ee567ab3aebaff2f08eb7ecd36b407" containerID="3c791398c8901cc967efdb129527c31d0f280075e23644fccab3f1df6a32a650" exitCode=0
Jul 29 16:51:15.998824 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:15.998664    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" event={"ID":"d5ee567ab3aebaff2f08eb7ecd36b407","Type":"ContainerDied","Data":"3c791398c8901cc967efdb129527c31d0f280075e23644fccab3f1df6a32a650"}
Jul 29 16:51:16.907465 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:16.907404    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:16.907960 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:16.907534    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:17.018772 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:17.018025    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" event={"ID":"d5ee567ab3aebaff2f08eb7ecd36b407","Type":"ContainerStarted","Data":"3d7dd9c4b2d6252e72dc508d7866977ff8e42be82d0778c430fb9db5f51f3060"}
Jul 29 16:51:17.547985 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:17.547007    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qd5\" (UniqueName: \"kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5\") pod \"network-check-target-zb222\" (UID: \"335e89f7-203c-4d41-8d24-b2fde7765514\") " pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:17.547985 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:17.547093    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:17.547985 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:17.547141    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs\") pod \"network-metrics-daemon-pmqs8\" (UID: \"438c3501-4812-489f-ae51-a25e01b9aacc\") " pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:17.547985 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:17.547301    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:17.547985 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:17.547362    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs podName:438c3501-4812-489f-ae51-a25e01b9aacc nodeName:}" failed. No retries permitted until 2026-07-29 16:51:21.547343922 +0000 UTC m=+10.036863574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs") pod "network-metrics-daemon-pmqs8" (UID: "438c3501-4812-489f-ae51-a25e01b9aacc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:17.547985 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:17.547776    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 16:51:17.547985 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:17.547795    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 16:51:17.547985 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:17.547808    2570 projected.go:196] Error preparing data for projected volume kube-api-access-97qd5 for pod openshift-network-diagnostics/network-check-target-zb222: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:17.547985 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:17.547853    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5 podName:335e89f7-203c-4d41-8d24-b2fde7765514 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:21.547838389 +0000 UTC m=+10.037358038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-97qd5" (UniqueName: "kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5") pod "network-check-target-zb222" (UID: "335e89f7-203c-4d41-8d24-b2fde7765514") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:17.547985 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:17.547917    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:17.547985 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:17.547950    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret podName:b1bd992f-213e-46bd-87a6-b06f7895719d nodeName:}" failed. No retries permitted until 2026-07-29 16:51:21.547939592 +0000 UTC m=+10.037459231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret") pod "global-pull-secret-syncer-dxzh5" (UID: "b1bd992f-213e-46bd-87a6-b06f7895719d") : object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:17.909639 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:17.909561    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:17.910052 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:17.909733    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:17.910191 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:17.910155    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:17.910311 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:17.910287    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:18.908074 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:18.908041    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:18.908256 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:18.908199    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:19.911496 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:19.909909    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:19.911496 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:19.910034    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:19.911496 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:19.910431    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:19.911496 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:19.910531    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:20.907332 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:20.907290    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:20.907582 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:20.907424    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:21.583427 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:21.583390    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qd5\" (UniqueName: \"kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5\") pod \"network-check-target-zb222\" (UID: \"335e89f7-203c-4d41-8d24-b2fde7765514\") " pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:21.583905 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:21.583456    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:21.583905 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:21.583497    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs\") pod \"network-metrics-daemon-pmqs8\" (UID: \"438c3501-4812-489f-ae51-a25e01b9aacc\") " pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:21.583905 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:21.583554    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 16:51:21.583905 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:21.583579    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 16:51:21.583905 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:21.583594    2570 projected.go:196] Error preparing data for projected volume kube-api-access-97qd5 for pod openshift-network-diagnostics/network-check-target-zb222: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:21.583905 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:21.583609    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:21.583905 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:21.583654    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5 podName:335e89f7-203c-4d41-8d24-b2fde7765514 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:29.583634654 +0000 UTC m=+18.073154290 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-97qd5" (UniqueName: "kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5") pod "network-check-target-zb222" (UID: "335e89f7-203c-4d41-8d24-b2fde7765514") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:21.583905 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:21.583677    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs podName:438c3501-4812-489f-ae51-a25e01b9aacc nodeName:}" failed. No retries permitted until 2026-07-29 16:51:29.583665522 +0000 UTC m=+18.073185162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs") pod "network-metrics-daemon-pmqs8" (UID: "438c3501-4812-489f-ae51-a25e01b9aacc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:21.583905 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:21.583702    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:21.583905 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:21.583745    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret podName:b1bd992f-213e-46bd-87a6-b06f7895719d nodeName:}" failed. No retries permitted until 2026-07-29 16:51:29.583722991 +0000 UTC m=+18.073242629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret") pod "global-pull-secret-syncer-dxzh5" (UID: "b1bd992f-213e-46bd-87a6-b06f7895719d") : object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:21.909347 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:21.909264    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:21.909512 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:21.909387    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:21.911553 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:21.911529    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:21.911674 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:21.911653    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:22.908727 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:22.908235    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:22.908727 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:22.908363    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:23.908193 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:23.908146    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:23.908344 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:23.908270    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:23.908344 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:23.908330    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:23.908441 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:23.908424    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:24.907981 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:24.907941    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:24.908415 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:24.908071    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:25.907353 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:25.907320    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:25.907502 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:25.907320    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:25.907502 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:25.907429    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:25.907502 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:25.907486    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:26.907450 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:26.907406    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:26.907883 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:26.907530    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:27.907430 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:27.907394    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:27.907606 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:27.907401    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:27.907606 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:27.907531    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:27.907952 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:27.907614    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:28.907411 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:28.907379    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:28.907609 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:28.907521    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:29.642755 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:29.642709    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qd5\" (UniqueName: \"kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5\") pod \"network-check-target-zb222\" (UID: \"335e89f7-203c-4d41-8d24-b2fde7765514\") " pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:29.643227 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:29.642776    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:29.643227 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:29.642820    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs\") pod \"network-metrics-daemon-pmqs8\" (UID: \"438c3501-4812-489f-ae51-a25e01b9aacc\") " pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:29.643227 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:29.642949    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:29.643227 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:29.642991    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:29.643227 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:29.643023    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret podName:b1bd992f-213e-46bd-87a6-b06f7895719d nodeName:}" failed. No retries permitted until 2026-07-29 16:51:45.643002171 +0000 UTC m=+34.132521809 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret") pod "global-pull-secret-syncer-dxzh5" (UID: "b1bd992f-213e-46bd-87a6-b06f7895719d") : object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:29.643227 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:29.642952    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 16:51:29.643227 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:29.643051    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs podName:438c3501-4812-489f-ae51-a25e01b9aacc nodeName:}" failed. No retries permitted until 2026-07-29 16:51:45.643034944 +0000 UTC m=+34.132554601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs") pod "network-metrics-daemon-pmqs8" (UID: "438c3501-4812-489f-ae51-a25e01b9aacc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:29.643227 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:29.643067    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 16:51:29.643227 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:29.643083    2570 projected.go:196] Error preparing data for projected volume kube-api-access-97qd5 for pod openshift-network-diagnostics/network-check-target-zb222: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:29.643227 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:29.643127    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5 podName:335e89f7-203c-4d41-8d24-b2fde7765514 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:45.643113885 +0000 UTC m=+34.132633533 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-97qd5" (UniqueName: "kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5") pod "network-check-target-zb222" (UID: "335e89f7-203c-4d41-8d24-b2fde7765514") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:29.908103 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:29.908018    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:29.908282 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:29.908146    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:29.908282 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:29.908195    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:29.908384 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:29.908284    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:30.907490 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:30.907454    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:30.907930 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:30.907583    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:31.909040 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:31.908993    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:31.909518 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:31.909103    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:31.909518 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:31.909136    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:31.909518 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:31.909207    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:32.907708 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:32.907668    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:32.907891 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:32.907799    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:33.908056 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:33.908031    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:33.908475 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:33.908201    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:33.909032 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:33.908750    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:33.909032 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:33.908855    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:34.050423 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.050190    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dq655" event={"ID":"8ed06670-f24d-4521-815a-0d27741edac3","Type":"ContainerStarted","Data":"ff77b20b7621c829b875765b31874c0d16bf04f0041f8e7a9ac9869f6710c141"}
Jul 29 16:51:34.051724 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.051694    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6qp9p" event={"ID":"e22624ca-991d-4e4d-8db7-02119caf746d","Type":"ContainerStarted","Data":"ebe133ab066c3b40cdf0e05b7e4a4809163fbee0efd68da498a01882367d423d"}
Jul 29 16:51:34.053327 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.053297    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq" event={"ID":"9243e853-a0da-492f-b653-b83b2d974747","Type":"ContainerStarted","Data":"a182987ddc2575b2073cafea2af313fc708ed1f6d903de341c382065c3e3eeee"}
Jul 29 16:51:34.054790 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.054767    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qzk6w" event={"ID":"ea15cf35-0332-4188-8b22-c4f58a5b9a9f","Type":"ContainerStarted","Data":"282074ccc2bd7ef38b9616de8a7b0167e6c259d72caa8b177e01705bf0d80456"}
Jul 29 16:51:34.056402 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.056380    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" event={"ID":"159881cacdca307468786a6d29c1c03f","Type":"ContainerStarted","Data":"3849627dfe85e99b73230b40fd9f93d0e89b1803a9011d5e24b7a8b4a98ef235"}
Jul 29 16:51:34.058033 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.058010    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-87796" event={"ID":"0e13b680-aa89-4a92-8ff2-1eb3e0a0309b","Type":"ContainerStarted","Data":"3979e09923420d30d823f704e4c656600a0be540ec8d4afce67bd19b1f06293b"}
Jul 29 16:51:34.059596 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.059563    2570 generic.go:356] "Generic (PLEG): container finished" podID="e356ec32-854a-4b20-985b-e93890a39aa3" containerID="7693cdefe8c3fe0efcec2f65c4ad89280d8d235ef1004553057cced1483117c5" exitCode=0
Jul 29 16:51:34.059709 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.059643    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxjp" event={"ID":"e356ec32-854a-4b20-985b-e93890a39aa3","Type":"ContainerDied","Data":"7693cdefe8c3fe0efcec2f65c4ad89280d8d235ef1004553057cced1483117c5"}
Jul 29 16:51:34.063844 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.063818    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z" event={"ID":"f7b01bca-6c8a-48e4-be26-e754c4d9df3c","Type":"ContainerStarted","Data":"e67c582e4f6adfb662119d04aaedb3b956e100ec9f143e115ee783a221830acd"}
Jul 29 16:51:34.066071 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.065847    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ttg74" event={"ID":"f28253eb-90bd-48ad-ade7-a48e9fab2605","Type":"ContainerStarted","Data":"26213f3185efb894138ad2a56f889e26d1aa8c4f5ff74b4a9bac9855f256bdee"}
Jul 29 16:51:34.066798 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.066754    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dq655" podStartSLOduration=2.448821205 podStartE2EDuration="22.066739544s" totalImagesPullingTime="19.617918339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:34.066500281 +0000 UTC m=+22.556019939" watchObservedRunningTime="2026-07-29 16:51:34.066739544 +0000 UTC m=+22.556259199"
Jul 29 16:51:34.066907 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.066832    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" podStartSLOduration=21.066826017 podStartE2EDuration="21.066826017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:17.042346069 +0000 UTC m=+5.531865725" watchObservedRunningTime="2026-07-29 16:51:34.066826017 +0000 UTC m=+22.556345673"
Jul 29 16:51:34.081720 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.081662    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6qp9p" podStartSLOduration=2.462492605 podStartE2EDuration="22.081640818s" totalImagesPullingTime="19.619148213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:34.081629625 +0000 UTC m=+22.571149285" watchObservedRunningTime="2026-07-29 16:51:34.081640818 +0000 UTC m=+22.571160469"
Jul 29 16:51:34.098849 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.098786    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qzk6w" podStartSLOduration=3.441865581 podStartE2EDuration="23.098767201s" totalImagesPullingTime="19.65690162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:34.098644807 +0000 UTC m=+22.588164463" watchObservedRunningTime="2026-07-29 16:51:34.098767201 +0000 UTC m=+22.588286858"
Jul 29 16:51:34.118385 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.118321    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-87796" podStartSLOduration=2.448255363 podStartE2EDuration="22.118300758s" totalImagesPullingTime="19.670045395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:34.117690072 +0000 UTC m=+22.607209727" watchObservedRunningTime="2026-07-29 16:51:34.118300758 +0000 UTC m=+22.607820415"
Jul 29 16:51:34.137442 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.137394    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ttg74" podStartSLOduration=10.191801505 podStartE2EDuration="22.137378358s" totalImagesPullingTime="11.945576853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:34.136901709 +0000 UTC m=+22.626421368" watchObservedRunningTime="2026-07-29 16:51:34.137378358 +0000 UTC m=+22.626898014"
Jul 29 16:51:34.187492 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.187451    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" podStartSLOduration=21.187436395 podStartE2EDuration="21.187436395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:34.155692082 +0000 UTC m=+22.645211738" watchObservedRunningTime="2026-07-29 16:51:34.187436395 +0000 UTC m=+22.676956077"
Jul 29 16:51:34.907934 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:34.907903    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:34.908113 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:34.908005    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:35.071618 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:35.071586    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq" event={"ID":"9243e853-a0da-492f-b653-b83b2d974747","Type":"ContainerStarted","Data":"4b73ecc03f5d34b2eef758ec89d73e8eb7f2c9cf8e0020f41211a2affc665667"}
Jul 29 16:51:35.071818 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:35.071659    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq" event={"ID":"9243e853-a0da-492f-b653-b83b2d974747","Type":"ContainerStarted","Data":"1d3049264b5c73cfbdfca728a27bfb832651f2a2eafd5354d339294a75affc2e"}
Jul 29 16:51:35.071818 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:35.071700    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq" event={"ID":"9243e853-a0da-492f-b653-b83b2d974747","Type":"ContainerStarted","Data":"3dbfa36bd76388d66388c8ecdd34a12611b7a70d52e8a0c3a61253147836f7dd"}
Jul 29 16:51:35.071818 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:35.071732    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq" event={"ID":"9243e853-a0da-492f-b653-b83b2d974747","Type":"ContainerStarted","Data":"d5f80748ff5d6bfba74e037a7eca4f2e3fe8fd642a26fab7b7bf7dcc4554efb5"}
Jul 29 16:51:35.071818 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:35.071764    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq" event={"ID":"9243e853-a0da-492f-b653-b83b2d974747","Type":"ContainerStarted","Data":"397bc9f96667494e27f138ca71b5f27906c5c2d8d36736d3e543fbc42f5781cd"}
Jul 29 16:51:35.073112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:35.073083    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7p58p" event={"ID":"4d4a05de-792b-4584-8f29-8f396f5f59fb","Type":"ContainerStarted","Data":"da82067ada43fd309b2f080c0aecf8eac97ce826adcf5d63502d56c5bc5e7750"}
Jul 29 16:51:35.088675 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:35.088625    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7p58p" podStartSLOduration=3.483835613 podStartE2EDuration="23.088607676s" totalImagesPullingTime="19.604772063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:35.088348357 +0000 UTC m=+23.577868012" watchObservedRunningTime="2026-07-29 16:51:35.088607676 +0000 UTC m=+23.578127332"
Jul 29 16:51:35.592947 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:35.592884    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ttg74"
Jul 29 16:51:35.593517 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:35.593499    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ttg74"
Jul 29 16:51:35.907709 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:35.907627    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:35.907852 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:35.907779    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:35.907852 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:35.907829    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:35.907945 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:35.907916    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:36.075581 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:36.075543    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ttg74"
Jul 29 16:51:36.076262 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:36.076240    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ttg74"
Jul 29 16:51:36.191545 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:36.191507    2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 16:51:36.873747 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:36.873584    2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T16:51:36.19152649Z","UUID":"544e850e-82a7-4d4f-b187-a8bda64a369e","Handler":null,"Name":"","Endpoint":""}
Jul 29 16:51:36.875337 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:36.875315    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 16:51:36.875467 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:36.875352    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 16:51:36.908250 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:36.908200    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:36.908428 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:36.908341    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:37.081832 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:37.081788    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq" event={"ID":"9243e853-a0da-492f-b653-b83b2d974747","Type":"ContainerStarted","Data":"1325347c0723945dc41cb68986533da88884a57952dab8aa5b937b1af38b2d4d"}
Jul 29 16:51:37.084386 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:37.084130    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z" event={"ID":"f7b01bca-6c8a-48e4-be26-e754c4d9df3c","Type":"ContainerStarted","Data":"75b55b3e0466e5126d8ff903ec832278c6556e0e1da6f9ed2b6a638a9fd1017d"}
Jul 29 16:51:37.908086 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:37.908050    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:37.908292 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:37.908100    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:37.908292 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:37.908203    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:37.908415 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:37.908297    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:38.907323 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:38.907146    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:38.907840 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:38.907426    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:39.094562 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:39.094527    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z" event={"ID":"f7b01bca-6c8a-48e4-be26-e754c4d9df3c","Type":"ContainerStarted","Data":"75cff8a99b4fa1c2e9a9bd5011e40fc34f5e854f5e3fba0a7d68857c4a2ddfcd"}
Jul 29 16:51:39.098400 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:39.098373    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq" event={"ID":"9243e853-a0da-492f-b653-b83b2d974747","Type":"ContainerStarted","Data":"b55c627130c5417075876290df5dfa292213a1ed9244a104c906aefa4368990f"}
Jul 29 16:51:39.098672 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:39.098646    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:39.098750 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:39.098724    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:39.098788 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:39.098771    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:39.115052 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:39.114731    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxr4z" podStartSLOduration=3.68371654 podStartE2EDuration="28.114666405s" totalImagesPullingTime="24.430949865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:39.113752979 +0000 UTC m=+27.603272635" watchObservedRunningTime="2026-07-29 16:51:39.114666405 +0000 UTC m=+27.604186062"
Jul 29 16:51:39.118150 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:39.118095    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:39.120403 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:39.120370    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:51:39.145101 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:39.145037    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq" podStartSLOduration=7.449607596 podStartE2EDuration="27.145014985s" totalImagesPullingTime="19.695407389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:39.14380135 +0000 UTC m=+27.633321006" watchObservedRunningTime="2026-07-29 16:51:39.145014985 +0000 UTC m=+27.634534641"
Jul 29 16:51:39.907927 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:39.907745    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:39.908673 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:39.907741    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:39.908673 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:39.908027    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:39.908673 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:39.908100    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:40.102455 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:40.102426    2570 generic.go:356] "Generic (PLEG): container finished" podID="e356ec32-854a-4b20-985b-e93890a39aa3" containerID="475187deb635f8c6cf976c2982fdc78d30fdcc77ef24fff98d00be044dca3c68" exitCode=0
Jul 29 16:51:40.102622 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:40.102494    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxjp" event={"ID":"e356ec32-854a-4b20-985b-e93890a39aa3","Type":"ContainerDied","Data":"475187deb635f8c6cf976c2982fdc78d30fdcc77ef24fff98d00be044dca3c68"}
Jul 29 16:51:40.907277 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:40.907237    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:40.907561 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:40.907358    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:41.225865 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:41.225830    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pmqs8"]
Jul 29 16:51:41.226297 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:41.226023    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:41.226297 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:41.226155    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:41.228111 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:41.228079    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zb222"]
Jul 29 16:51:41.228261 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:41.228212    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:41.228366 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:41.228339    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:41.231722 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:41.231697    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dxzh5"]
Jul 29 16:51:41.231831 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:41.231798    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:41.231937 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:41.231915    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:42.109653 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:42.109622    2570 generic.go:356] "Generic (PLEG): container finished" podID="e356ec32-854a-4b20-985b-e93890a39aa3" containerID="a543fe8144940676f9a63582778887a8dca2894763c86751fde4c4e34c365fe7" exitCode=0
Jul 29 16:51:42.109829 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:42.109663    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxjp" event={"ID":"e356ec32-854a-4b20-985b-e93890a39aa3","Type":"ContainerDied","Data":"a543fe8144940676f9a63582778887a8dca2894763c86751fde4c4e34c365fe7"}
Jul 29 16:51:42.908279 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:42.908244    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:42.908745 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:42.908284    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:42.908745 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:42.908244    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:42.908745 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:42.908379    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:42.908745 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:42.908455    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:42.908745 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:42.908524    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:44.116315 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:44.116282    2570 generic.go:356] "Generic (PLEG): container finished" podID="e356ec32-854a-4b20-985b-e93890a39aa3" containerID="d497db1c514b0b5c7cdd4a9626d99ee421ea5a37172c262b95e366b9b6e31caa" exitCode=0
Jul 29 16:51:44.116720 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:44.116333    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxjp" event={"ID":"e356ec32-854a-4b20-985b-e93890a39aa3","Type":"ContainerDied","Data":"d497db1c514b0b5c7cdd4a9626d99ee421ea5a37172c262b95e366b9b6e31caa"}
Jul 29 16:51:44.907875 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:44.907668    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:44.908054 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:44.907715    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:44.908054 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:44.907961    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:44.908054 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:44.907727    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:44.908054 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:44.908033    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:44.908285 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:44.908152    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:45.661098 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:45.661057    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qd5\" (UniqueName: \"kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5\") pod \"network-check-target-zb222\" (UID: \"335e89f7-203c-4d41-8d24-b2fde7765514\") " pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:45.661543 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:45.661135    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:45.661543 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:45.661199    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs\") pod \"network-metrics-daemon-pmqs8\" (UID: \"438c3501-4812-489f-ae51-a25e01b9aacc\") " pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:45.661543 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:45.661207    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 16:51:45.661543 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:45.661239    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 16:51:45.661543 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:45.661250    2570 projected.go:196] Error preparing data for projected volume kube-api-access-97qd5 for pod openshift-network-diagnostics/network-check-target-zb222: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:45.661543 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:45.661285    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:45.661543 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:45.661290    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:45.661543 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:45.661305    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5 podName:335e89f7-203c-4d41-8d24-b2fde7765514 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:17.661289264 +0000 UTC m=+66.150808919 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-97qd5" (UniqueName: "kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5") pod "network-check-target-zb222" (UID: "335e89f7-203c-4d41-8d24-b2fde7765514") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:45.661543 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:45.661334    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs podName:438c3501-4812-489f-ae51-a25e01b9aacc nodeName:}" failed. No retries permitted until 2026-07-29 16:52:17.661321039 +0000 UTC m=+66.150840674 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs") pod "network-metrics-daemon-pmqs8" (UID: "438c3501-4812-489f-ae51-a25e01b9aacc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:45.661543 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:45.661354    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret podName:b1bd992f-213e-46bd-87a6-b06f7895719d nodeName:}" failed. No retries permitted until 2026-07-29 16:52:17.661343208 +0000 UTC m=+66.150862845 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret") pod "global-pull-secret-syncer-dxzh5" (UID: "b1bd992f-213e-46bd-87a6-b06f7895719d") : object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:46.908239 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:46.908203    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:46.908239 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:46.908225    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:46.908239 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:46.908245    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:46.908767 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:46.908335    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxzh5" podUID="b1bd992f-213e-46bd-87a6-b06f7895719d"
Jul 29 16:51:46.908767 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:46.908420    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pmqs8" podUID="438c3501-4812-489f-ae51-a25e01b9aacc"
Jul 29 16:51:46.908767 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:46.908498    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb222" podUID="335e89f7-203c-4d41-8d24-b2fde7765514"
Jul 29 16:51:47.412710 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.412679    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeReady"
Jul 29 16:51:47.413374 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.413344    2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 16:51:47.463304 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.463266    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn"]
Jul 29 16:51:47.468323 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.468291    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57b8589c7d-4x49t"]
Jul 29 16:51:47.468479 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.468428    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn"
Jul 29 16:51:47.471793 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.471688    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 29 16:51:47.471793 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.471753    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-hg8mz\""
Jul 29 16:51:47.471793 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.471768    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 29 16:51:47.472036 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.471787    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 29 16:51:47.472036 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.472000    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 29 16:51:47.472899 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.472878    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5mnw6"]
Jul 29 16:51:47.473047 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.473012    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.476216 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.476191    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 16:51:47.476503 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.476483    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 16:51:47.476783 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.476726    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-78qw7\""
Jul 29 16:51:47.476870 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.476850    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"]
Jul 29 16:51:47.476946 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.476931    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 16:51:47.477006 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.476983    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5mnw6"
Jul 29 16:51:47.480464 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.480443    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-gg4gt"]
Jul 29 16:51:47.480580 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.480570    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"
Jul 29 16:51:47.484201 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.484148    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-w2v54\""
Jul 29 16:51:47.484885 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.484664    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 29 16:51:47.484885 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.484676    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cwhbz\""
Jul 29 16:51:47.485041 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.484913    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm"]
Jul 29 16:51:47.485041 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.484993    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 29 16:51:47.485041 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.485033    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 29 16:51:47.485190 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.485141    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 29 16:51:47.485297 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.485276    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 29 16:51:47.485610 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.485593    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:47.487559 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.487507    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 16:51:47.489903 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.489232    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"]
Jul 29 16:51:47.489903 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.489471    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm"
Jul 29 16:51:47.490942 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.490645    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 29 16:51:47.490942 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.490800    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 29 16:51:47.490942 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.490824    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 29 16:51:47.490942 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.490849    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-72jrt\""
Jul 29 16:51:47.491225 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.490979    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 29 16:51:47.492697 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.492676    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"]
Jul 29 16:51:47.492807 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.492717    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn"]
Jul 29 16:51:47.492807 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.492738    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57b8589c7d-4x49t"]
Jul 29 16:51:47.492807 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.492789    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:51:47.493360 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.493341    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 29 16:51:47.493608 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.493396    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6pm5r\""
Jul 29 16:51:47.493693 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.493401    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 29 16:51:47.493994 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.493872    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 29 16:51:47.493994 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.493876    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 29 16:51:47.495089 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.495069    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 16:51:47.495451 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.495429    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-v8hks\""
Jul 29 16:51:47.495881 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.495616    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 29 16:51:47.497210 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.497188    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 29 16:51:47.501548 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.501523    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 16:51:47.510827 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.506423    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-gg4gt"]
Jul 29 16:51:47.510827 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.508563    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-95x46"]
Jul 29 16:51:47.512983 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.512753    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 29 16:51:47.514783 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.514435    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-zx4jr"]
Jul 29 16:51:47.514783 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.514516    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:47.517365 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.517345    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 16:51:47.517789 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.517770    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-tv5bc"]
Jul 29 16:51:47.517901 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.517883    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.517962 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.517901    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2hb68\""
Jul 29 16:51:47.517962 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.517801    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 16:51:47.521071 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.521045    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 16:51:47.521299 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.521281    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-q7g5h\""
Jul 29 16:51:47.521448 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.521402    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 29 16:51:47.521774 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.521750    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5df7c8cd4c-mgz6p"]
Jul 29 16:51:47.521858 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.521803    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 29 16:51:47.521858 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.521842    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tv5bc"
Jul 29 16:51:47.522354 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.522334    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 16:51:47.524956 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.524939    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gqw66\""
Jul 29 16:51:47.525210 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.525151    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5mnw6"]
Jul 29 16:51:47.525300 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.525213    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"]
Jul 29 16:51:47.525300 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.525273    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 16:51:47.525300 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.525296    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:47.525525 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.525507    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 16:51:47.528252 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.528234    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 29 16:51:47.531403 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.531384    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 29 16:51:47.531688 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.531673    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 29 16:51:47.533677 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.533507    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 29 16:51:47.533775 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.533723    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-zx4jr"]
Jul 29 16:51:47.534083 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.534066    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 16:51:47.534343 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.534322    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 29 16:51:47.534417 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.534234    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 16:51:47.534483 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.534185    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jhsnv\""
Jul 29 16:51:47.534617 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.534599    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm"]
Jul 29 16:51:47.536107 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.536086    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-95x46"]
Jul 29 16:51:47.536958 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.536927    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-tv5bc"]
Jul 29 16:51:47.552663 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.552640    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5df7c8cd4c-mgz6p"]
Jul 29 16:51:47.571519 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.571494    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zl259"]
Jul 29 16:51:47.575363 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575335    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zl259"
Jul 29 16:51:47.575490 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575468    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/357c4ea3-b7a7-4f23-8ac7-f06745febc90-trusted-ca\") pod \"console-operator-584dcb6b54-gg4gt\" (UID: \"357c4ea3-b7a7-4f23-8ac7-f06745febc90\") " pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:47.575564 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575502    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c5076ed-8a8a-4866-801a-5b0037ec4d36-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7httm\" (UID: \"2c5076ed-8a8a-4866-801a-5b0037ec4d36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm"
Jul 29 16:51:47.575564 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575531    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2c5076ed-8a8a-4866-801a-5b0037ec4d36-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7httm\" (UID: \"2c5076ed-8a8a-4866-801a-5b0037ec4d36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm"
Jul 29 16:51:47.575658 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575557    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a5e4084-948e-4a3b-9752-b80d169c5078-trusted-ca\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.575658 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575587    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1a5e4084-948e-4a3b-9752-b80d169c5078-installation-pull-secrets\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.575658 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575643    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a86ddea8-14e1-4ea5-bd16-20e267ef74f1-config\") pod \"service-ca-operator-5d6c75f84f-fb7nn\" (UID: \"a86ddea8-14e1-4ea5-bd16-20e267ef74f1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn"
Jul 29 16:51:47.575779 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575750    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-certificates\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.575815 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575780    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvx47\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-kube-api-access-hvx47\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.575852 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575821    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-bound-sa-token\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.575884 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575847    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a86ddea8-14e1-4ea5-bd16-20e267ef74f1-serving-cert\") pod \"service-ca-operator-5d6c75f84f-fb7nn\" (UID: \"a86ddea8-14e1-4ea5-bd16-20e267ef74f1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn"
Jul 29 16:51:47.575884 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575877    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf8p2\" (UniqueName: \"kubernetes.io/projected/2c5076ed-8a8a-4866-801a-5b0037ec4d36-kube-api-access-lf8p2\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7httm\" (UID: \"2c5076ed-8a8a-4866-801a-5b0037ec4d36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm"
Jul 29 16:51:47.575971 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575901    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqf8h\" (UniqueName: \"kubernetes.io/projected/a86ddea8-14e1-4ea5-bd16-20e267ef74f1-kube-api-access-rqf8h\") pod \"service-ca-operator-5d6c75f84f-fb7nn\" (UID: \"a86ddea8-14e1-4ea5-bd16-20e267ef74f1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn"
Jul 29 16:51:47.575971 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575915    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/357c4ea3-b7a7-4f23-8ac7-f06745febc90-config\") pod \"console-operator-584dcb6b54-gg4gt\" (UID: \"357c4ea3-b7a7-4f23-8ac7-f06745febc90\") " pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:47.576071 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.575992    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1a5e4084-948e-4a3b-9752-b80d169c5078-ca-trust-extracted\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.576071 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.576025    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w8bc\" (UniqueName: \"kubernetes.io/projected/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-kube-api-access-7w8bc\") pod \"cluster-samples-operator-6748489656-stpc7\" (UID: \"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"
Jul 29 16:51:47.576071 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.576051    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd8cq\" (UniqueName: \"kubernetes.io/projected/50f45c1b-30e6-4965-8174-24bd4e0f815b-kube-api-access-sd8cq\") pod \"volume-data-source-validator-549c6d54cf-5mnw6\" (UID: \"50f45c1b-30e6-4965-8174-24bd4e0f815b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5mnw6"
Jul 29 16:51:47.576201 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.576145    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.576201 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.576189    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtfmt\" (UniqueName: \"kubernetes.io/projected/8d895e18-c64e-4656-868b-4d5191988051-kube-api-access-vtfmt\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:51:47.576293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.576214    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1a5e4084-948e-4a3b-9752-b80d169c5078-image-registry-private-configuration\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.576293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.576232    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:51:47.576293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.576248    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/357c4ea3-b7a7-4f23-8ac7-f06745febc90-serving-cert\") pod \"console-operator-584dcb6b54-gg4gt\" (UID: \"357c4ea3-b7a7-4f23-8ac7-f06745febc90\") " pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:47.576431 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.576298    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-stpc7\" (UID: \"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"
Jul 29 16:51:47.576431 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.576341    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8d895e18-c64e-4656-868b-4d5191988051-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:51:47.576431 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.576369    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf4r4\" (UniqueName: \"kubernetes.io/projected/357c4ea3-b7a7-4f23-8ac7-f06745febc90-kube-api-access-kf4r4\") pod \"console-operator-584dcb6b54-gg4gt\" (UID: \"357c4ea3-b7a7-4f23-8ac7-f06745febc90\") " pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:47.578508 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.578488    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 16:51:47.578776 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.578751    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 16:51:47.578776 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.578751    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 16:51:47.578931 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.578757    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dg98s\""
Jul 29 16:51:47.590957 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.590929    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zl259"]
Jul 29 16:51:47.677305 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677272    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-bound-sa-token\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.677305 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677309    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:47.677550 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677341    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvx47\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-kube-api-access-hvx47\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.677550 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677365    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-default-certificate\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:47.677550 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677391    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a86ddea8-14e1-4ea5-bd16-20e267ef74f1-serving-cert\") pod \"service-ca-operator-5d6c75f84f-fb7nn\" (UID: \"a86ddea8-14e1-4ea5-bd16-20e267ef74f1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn"
Jul 29 16:51:47.677550 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677415    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-service-ca-bundle\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.677550 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677439    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snjb5\" (UniqueName: \"kubernetes.io/projected/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-kube-api-access-snjb5\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:47.677550 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677521    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:47.677862 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677670    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/357c4ea3-b7a7-4f23-8ac7-f06745febc90-config\") pod \"console-operator-584dcb6b54-gg4gt\" (UID: \"357c4ea3-b7a7-4f23-8ac7-f06745febc90\") " pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:47.677862 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677710    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.677862 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677742    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w8bc\" (UniqueName: \"kubernetes.io/projected/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-kube-api-access-7w8bc\") pod \"cluster-samples-operator-6748489656-stpc7\" (UID: \"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"
Jul 29 16:51:47.677862 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677769    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8cq\" (UniqueName: \"kubernetes.io/projected/50f45c1b-30e6-4965-8174-24bd4e0f815b-kube-api-access-sd8cq\") pod \"volume-data-source-validator-549c6d54cf-5mnw6\" (UID: \"50f45c1b-30e6-4965-8174-24bd4e0f815b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5mnw6"
Jul 29 16:51:47.677862 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677812    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-config-volume\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:47.677862 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677845    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/357c4ea3-b7a7-4f23-8ac7-f06745febc90-trusted-ca\") pod \"console-operator-584dcb6b54-gg4gt\" (UID: \"357c4ea3-b7a7-4f23-8ac7-f06745febc90\") " pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:47.678345 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677874    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-stpc7\" (UID: \"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"
Jul 29 16:51:47.678345 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677898    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:47.678345 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677926    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8d895e18-c64e-4656-868b-4d5191988051-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:51:47.678345 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677956    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2c5076ed-8a8a-4866-801a-5b0037ec4d36-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7httm\" (UID: \"2c5076ed-8a8a-4866-801a-5b0037ec4d36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm"
Jul 29 16:51:47.678345 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.677981    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-stats-auth\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:47.678345 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.678007    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a5e4084-948e-4a3b-9752-b80d169c5078-trusted-ca\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.678345 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.678057    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1a5e4084-948e-4a3b-9752-b80d169c5078-installation-pull-secrets\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.678345 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.678083    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a86ddea8-14e1-4ea5-bd16-20e267ef74f1-config\") pod \"service-ca-operator-5d6c75f84f-fb7nn\" (UID: \"a86ddea8-14e1-4ea5-bd16-20e267ef74f1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn"
Jul 29 16:51:47.678345 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.678116    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/357c4ea3-b7a7-4f23-8ac7-f06745febc90-serving-cert\") pod \"console-operator-584dcb6b54-gg4gt\" (UID: \"357c4ea3-b7a7-4f23-8ac7-f06745febc90\") " pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:47.678782 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.678581    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/357c4ea3-b7a7-4f23-8ac7-f06745febc90-config\") pod \"console-operator-584dcb6b54-gg4gt\" (UID: \"357c4ea3-b7a7-4f23-8ac7-f06745febc90\") " pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:47.679051 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.679026    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/357c4ea3-b7a7-4f23-8ac7-f06745febc90-trusted-ca\") pod \"console-operator-584dcb6b54-gg4gt\" (UID: \"357c4ea3-b7a7-4f23-8ac7-f06745febc90\") " pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:47.679218 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.679075    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8d895e18-c64e-4656-868b-4d5191988051-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:51:47.679345 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:47.679088    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 16:51:47.679650 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:47.679634    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls podName:ba8efd0d-7b3c-4c24-9443-f7bc8d66b535 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:48.179613079 +0000 UTC m=+36.669132724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls") pod "cluster-samples-operator-6748489656-stpc7" (UID: "ba8efd0d-7b3c-4c24-9443-f7bc8d66b535") : secret "samples-operator-tls" not found
Jul 29 16:51:47.679792 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.679775    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf8p2\" (UniqueName: \"kubernetes.io/projected/2c5076ed-8a8a-4866-801a-5b0037ec4d36-kube-api-access-lf8p2\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7httm\" (UID: \"2c5076ed-8a8a-4866-801a-5b0037ec4d36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm"
Jul 29 16:51:47.679927 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.679909    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml2lx\" (UniqueName: \"kubernetes.io/projected/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-kube-api-access-ml2lx\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:47.680027 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.679947    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7859m\" (UniqueName: \"kubernetes.io/projected/240795ba-8cfb-4ab3-b9ac-6d453268441f-kube-api-access-7859m\") pod \"ingress-canary-zl259\" (UID: \"240795ba-8cfb-4ab3-b9ac-6d453268441f\") " pod="openshift-ingress-canary/ingress-canary-zl259"
Jul 29 16:51:47.680027 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.679974    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kthqm\" (UniqueName: \"kubernetes.io/projected/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-kube-api-access-kthqm\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.680027 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.679646    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2c5076ed-8a8a-4866-801a-5b0037ec4d36-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7httm\" (UID: \"2c5076ed-8a8a-4866-801a-5b0037ec4d36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm"
Jul 29 16:51:47.680027 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680005    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqf8h\" (UniqueName: \"kubernetes.io/projected/a86ddea8-14e1-4ea5-bd16-20e267ef74f1-kube-api-access-rqf8h\") pod \"service-ca-operator-5d6c75f84f-fb7nn\" (UID: \"a86ddea8-14e1-4ea5-bd16-20e267ef74f1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn"
Jul 29 16:51:47.680239 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680057    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/240795ba-8cfb-4ab3-b9ac-6d453268441f-cert\") pod \"ingress-canary-zl259\" (UID: \"240795ba-8cfb-4ab3-b9ac-6d453268441f\") " pod="openshift-ingress-canary/ingress-canary-zl259"
Jul 29 16:51:47.680239 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680091    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1a5e4084-948e-4a3b-9752-b80d169c5078-ca-trust-extracted\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.680239 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680116    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-serving-cert\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.680239 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680142    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.680239 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680192    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtfmt\" (UniqueName: \"kubernetes.io/projected/8d895e18-c64e-4656-868b-4d5191988051-kube-api-access-vtfmt\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:51:47.680239 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680219    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4vcs\" (UniqueName: \"kubernetes.io/projected/259db7d9-6ddd-4647-96d6-6782765aff7e-kube-api-access-m4vcs\") pod \"network-check-source-7d88cb55d8-tv5bc\" (UID: \"259db7d9-6ddd-4647-96d6-6782765aff7e\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tv5bc"
Jul 29 16:51:47.680522 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680249    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1a5e4084-948e-4a3b-9752-b80d169c5078-image-registry-private-configuration\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.680522 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680310    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:51:47.680522 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680354    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf4r4\" (UniqueName: \"kubernetes.io/projected/357c4ea3-b7a7-4f23-8ac7-f06745febc90-kube-api-access-kf4r4\") pod \"console-operator-584dcb6b54-gg4gt\" (UID: \"357c4ea3-b7a7-4f23-8ac7-f06745febc90\") " pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:47.680522 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680378    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-snapshots\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.680522 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680406    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c5076ed-8a8a-4866-801a-5b0037ec4d36-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7httm\" (UID: \"2c5076ed-8a8a-4866-801a-5b0037ec4d36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm"
Jul 29 16:51:47.680522 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680432    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-tmp-dir\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:47.680522 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.679699    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a86ddea8-14e1-4ea5-bd16-20e267ef74f1-config\") pod \"service-ca-operator-5d6c75f84f-fb7nn\" (UID: \"a86ddea8-14e1-4ea5-bd16-20e267ef74f1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn"
Jul 29 16:51:47.680522 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680466    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-certificates\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.680522 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.680493    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-tmp\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.680927 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:47.680688    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 16:51:47.680927 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:47.680781    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls podName:8d895e18-c64e-4656-868b-4d5191988051 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:48.180761807 +0000 UTC m=+36.670281445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-787s9" (UID: "8d895e18-c64e-4656-868b-4d5191988051") : secret "cluster-monitoring-operator-tls" not found
Jul 29 16:51:47.680927 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:47.680818    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 16:51:47.680927 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:47.680834    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8589c7d-4x49t: secret "image-registry-tls" not found
Jul 29 16:51:47.681115 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:47.680978    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls podName:1a5e4084-948e-4a3b-9752-b80d169c5078 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:48.180960145 +0000 UTC m=+36.670479782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls") pod "image-registry-57b8589c7d-4x49t" (UID: "1a5e4084-948e-4a3b-9752-b80d169c5078") : secret "image-registry-tls" not found
Jul 29 16:51:47.681611 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.681591    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1a5e4084-948e-4a3b-9752-b80d169c5078-ca-trust-extracted\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.681842 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.681812    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a5e4084-948e-4a3b-9752-b80d169c5078-trusted-ca\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.682069 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.682046    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-certificates\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.683189 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.683130    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/357c4ea3-b7a7-4f23-8ac7-f06745febc90-serving-cert\") pod \"console-operator-584dcb6b54-gg4gt\" (UID: \"357c4ea3-b7a7-4f23-8ac7-f06745febc90\") " pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:47.683263 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.683190    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1a5e4084-948e-4a3b-9752-b80d169c5078-installation-pull-secrets\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.683431 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.683407    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1a5e4084-948e-4a3b-9752-b80d169c5078-image-registry-private-configuration\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.683865 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.683847    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c5076ed-8a8a-4866-801a-5b0037ec4d36-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7httm\" (UID: \"2c5076ed-8a8a-4866-801a-5b0037ec4d36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm"
Jul 29 16:51:47.684461 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.684433    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a86ddea8-14e1-4ea5-bd16-20e267ef74f1-serving-cert\") pod \"service-ca-operator-5d6c75f84f-fb7nn\" (UID: \"a86ddea8-14e1-4ea5-bd16-20e267ef74f1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn"
Jul 29 16:51:47.689542 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.689515    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8cq\" (UniqueName: \"kubernetes.io/projected/50f45c1b-30e6-4965-8174-24bd4e0f815b-kube-api-access-sd8cq\") pod \"volume-data-source-validator-549c6d54cf-5mnw6\" (UID: \"50f45c1b-30e6-4965-8174-24bd4e0f815b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5mnw6"
Jul 29 16:51:47.689939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.689917    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-bound-sa-token\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.692361 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.692338    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvx47\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-kube-api-access-hvx47\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:47.695763 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.695729    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqf8h\" (UniqueName: \"kubernetes.io/projected/a86ddea8-14e1-4ea5-bd16-20e267ef74f1-kube-api-access-rqf8h\") pod \"service-ca-operator-5d6c75f84f-fb7nn\" (UID: \"a86ddea8-14e1-4ea5-bd16-20e267ef74f1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn"
Jul 29 16:51:47.696255 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.696232    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf4r4\" (UniqueName: \"kubernetes.io/projected/357c4ea3-b7a7-4f23-8ac7-f06745febc90-kube-api-access-kf4r4\") pod \"console-operator-584dcb6b54-gg4gt\" (UID: \"357c4ea3-b7a7-4f23-8ac7-f06745febc90\") " pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:47.696628 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.696609    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf8p2\" (UniqueName: \"kubernetes.io/projected/2c5076ed-8a8a-4866-801a-5b0037ec4d36-kube-api-access-lf8p2\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7httm\" (UID: \"2c5076ed-8a8a-4866-801a-5b0037ec4d36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm"
Jul 29 16:51:47.697125 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.697107    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w8bc\" (UniqueName: \"kubernetes.io/projected/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-kube-api-access-7w8bc\") pod \"cluster-samples-operator-6748489656-stpc7\" (UID: \"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"
Jul 29 16:51:47.699900 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.699877    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtfmt\" (UniqueName: \"kubernetes.io/projected/8d895e18-c64e-4656-868b-4d5191988051-kube-api-access-vtfmt\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:51:47.781131 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781039    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:47.781131 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781092    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-stats-auth\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:47.781382 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781145    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2lx\" (UniqueName: \"kubernetes.io/projected/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-kube-api-access-ml2lx\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:47.781382 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781189    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7859m\" (UniqueName: \"kubernetes.io/projected/240795ba-8cfb-4ab3-b9ac-6d453268441f-kube-api-access-7859m\") pod \"ingress-canary-zl259\" (UID: \"240795ba-8cfb-4ab3-b9ac-6d453268441f\") " pod="openshift-ingress-canary/ingress-canary-zl259"
Jul 29 16:51:47.781382 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781217    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kthqm\" (UniqueName: \"kubernetes.io/projected/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-kube-api-access-kthqm\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.781382 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:47.781241    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 16:51:47.781382 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:47.781322    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs podName:fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:48.281301085 +0000 UTC m=+36.770820739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs") pod "router-default-5df7c8cd4c-mgz6p" (UID: "fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6") : secret "router-metrics-certs-default" not found
Jul 29 16:51:47.781559 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781251    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/240795ba-8cfb-4ab3-b9ac-6d453268441f-cert\") pod \"ingress-canary-zl259\" (UID: \"240795ba-8cfb-4ab3-b9ac-6d453268441f\") " pod="openshift-ingress-canary/ingress-canary-zl259"
Jul 29 16:51:47.781559 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781457    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-serving-cert\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.781559 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781486    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4vcs\" (UniqueName: \"kubernetes.io/projected/259db7d9-6ddd-4647-96d6-6782765aff7e-kube-api-access-m4vcs\") pod \"network-check-source-7d88cb55d8-tv5bc\" (UID: \"259db7d9-6ddd-4647-96d6-6782765aff7e\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tv5bc"
Jul 29 16:51:47.781559 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781522    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-snapshots\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.781559 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781540    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-tmp-dir\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:47.781787 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781568    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-tmp\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.781787 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781596    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:47.781787 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781621    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-default-certificate\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:47.781787 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781657    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-service-ca-bundle\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.781787 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781682    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snjb5\" (UniqueName: \"kubernetes.io/projected/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-kube-api-access-snjb5\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:47.781787 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781706    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:47.782035 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:47.781823    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle podName:fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:48.281807036 +0000 UTC m=+36.771326671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle") pod "router-default-5df7c8cd4c-mgz6p" (UID: "fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6") : configmap references non-existent config key: service-ca.crt
Jul 29 16:51:47.782035 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:47.781890    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 16:51:47.782035 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:47.781929    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls podName:b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:48.281917921 +0000 UTC m=+36.771437572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls") pod "dns-default-95x46" (UID: "b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85") : secret "dns-default-metrics-tls" not found
Jul 29 16:51:47.782035 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.781968    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-tmp\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.782265 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.782244    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-tmp-dir\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:47.782355 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.782279    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.782355 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.782289    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-snapshots\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.782355 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.782328    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-config-volume\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:47.782649 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.782624    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-service-ca-bundle\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.782878 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.782834    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-config-volume\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:47.783119 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.783068    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.783632 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.783606    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn"
Jul 29 16:51:47.784177 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.784109    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-serving-cert\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.784308 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.784282    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-stats-auth\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:47.784593 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.784575    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/240795ba-8cfb-4ab3-b9ac-6d453268441f-cert\") pod \"ingress-canary-zl259\" (UID: \"240795ba-8cfb-4ab3-b9ac-6d453268441f\") " pod="openshift-ingress-canary/ingress-canary-zl259"
Jul 29 16:51:47.785368 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.785351    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-default-certificate\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:47.795006 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.794935    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snjb5\" (UniqueName: \"kubernetes.io/projected/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-kube-api-access-snjb5\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:47.795006 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.794940    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kthqm\" (UniqueName: \"kubernetes.io/projected/af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e-kube-api-access-kthqm\") pod \"insights-operator-5885fdb759-zx4jr\" (UID: \"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e\") " pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.796878 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.796822    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4vcs\" (UniqueName: \"kubernetes.io/projected/259db7d9-6ddd-4647-96d6-6782765aff7e-kube-api-access-m4vcs\") pod \"network-check-source-7d88cb55d8-tv5bc\" (UID: \"259db7d9-6ddd-4647-96d6-6782765aff7e\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tv5bc"
Jul 29 16:51:47.797260 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.797237    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7859m\" (UniqueName: \"kubernetes.io/projected/240795ba-8cfb-4ab3-b9ac-6d453268441f-kube-api-access-7859m\") pod \"ingress-canary-zl259\" (UID: \"240795ba-8cfb-4ab3-b9ac-6d453268441f\") " pod="openshift-ingress-canary/ingress-canary-zl259"
Jul 29 16:51:47.797910 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.797891    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2lx\" (UniqueName: \"kubernetes.io/projected/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-kube-api-access-ml2lx\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:47.806728 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.806700    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5mnw6"
Jul 29 16:51:47.826470 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.826437    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:47.835578 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.835550    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm"
Jul 29 16:51:47.858966 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.858931    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-zx4jr"
Jul 29 16:51:47.867122 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.867095    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tv5bc"
Jul 29 16:51:47.886406 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:47.886368    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zl259"
Jul 29 16:51:48.098240 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.097022    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"]
Jul 29 16:51:48.102000 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.100792    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"
Jul 29 16:51:48.113203 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.105894    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 29 16:51:48.113203 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.106188    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-njmsx\""
Jul 29 16:51:48.113203 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.106423    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 29 16:51:48.115411 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.114240    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"]
Jul 29 16:51:48.191260 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.187630    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-stpc7\" (UID: \"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"
Jul 29 16:51:48.191260 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.187699    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a232c9c-55d3-4118-90cc-49aae6db92d5-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-d59pv\" (UID: \"9a232c9c-55d3-4118-90cc-49aae6db92d5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"
Jul 29 16:51:48.191260 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.187787    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d59pv\" (UID: \"9a232c9c-55d3-4118-90cc-49aae6db92d5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"
Jul 29 16:51:48.191260 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.187820    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:48.191260 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.187855    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:51:48.191260 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.188064    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 16:51:48.191260 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.188132    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls podName:8d895e18-c64e-4656-868b-4d5191988051 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:49.188112722 +0000 UTC m=+37.677632379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-787s9" (UID: "8d895e18-c64e-4656-868b-4d5191988051") : secret "cluster-monitoring-operator-tls" not found
Jul 29 16:51:48.191260 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.188637    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 16:51:48.191260 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.188692    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls podName:ba8efd0d-7b3c-4c24-9443-f7bc8d66b535 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:49.188676843 +0000 UTC m=+37.678196496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls") pod "cluster-samples-operator-6748489656-stpc7" (UID: "ba8efd0d-7b3c-4c24-9443-f7bc8d66b535") : secret "samples-operator-tls" not found
Jul 29 16:51:48.191260 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.188781    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 16:51:48.191260 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.188791    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8589c7d-4x49t: secret "image-registry-tls" not found
Jul 29 16:51:48.191260 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.188822    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls podName:1a5e4084-948e-4a3b-9752-b80d169c5078 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:49.188811571 +0000 UTC m=+37.678331226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls") pod "image-registry-57b8589c7d-4x49t" (UID: "1a5e4084-948e-4a3b-9752-b80d169c5078") : secret "image-registry-tls" not found
Jul 29 16:51:48.290050 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.288763    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:48.290050 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.288980    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:48.290050 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.289039    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:48.290050 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.289064    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a232c9c-55d3-4118-90cc-49aae6db92d5-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-d59pv\" (UID: \"9a232c9c-55d3-4118-90cc-49aae6db92d5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"
Jul 29 16:51:48.290050 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.289142    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d59pv\" (UID: \"9a232c9c-55d3-4118-90cc-49aae6db92d5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"
Jul 29 16:51:48.290050 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.289272    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle podName:fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:49.28925181 +0000 UTC m=+37.778771457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle") pod "router-default-5df7c8cd4c-mgz6p" (UID: "fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6") : configmap references non-existent config key: service-ca.crt
Jul 29 16:51:48.290050 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.289319    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 16:51:48.290050 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.289338    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 16:51:48.290050 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.289324    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 16:51:48.290050 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.289370    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls podName:b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:49.289359996 +0000 UTC m=+37.778879630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls") pod "dns-default-95x46" (UID: "b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85") : secret "dns-default-metrics-tls" not found
Jul 29 16:51:48.290050 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.289424    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs podName:fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:49.289405414 +0000 UTC m=+37.778925054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs") pod "router-default-5df7c8cd4c-mgz6p" (UID: "fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6") : secret "router-metrics-certs-default" not found
Jul 29 16:51:48.290050 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.289441    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert podName:9a232c9c-55d3-4118-90cc-49aae6db92d5 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:48.789431639 +0000 UTC m=+37.278951274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d59pv" (UID: "9a232c9c-55d3-4118-90cc-49aae6db92d5") : secret "networking-console-plugin-cert" not found
Jul 29 16:51:48.290050 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.289981    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a232c9c-55d3-4118-90cc-49aae6db92d5-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-d59pv\" (UID: \"9a232c9c-55d3-4118-90cc-49aae6db92d5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"
Jul 29 16:51:48.348716 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.348687    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-zx4jr"]
Jul 29 16:51:48.352025 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.351992    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-tv5bc"]
Jul 29 16:51:48.353832 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.353810    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5mnw6"]
Jul 29 16:51:48.356860 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:48.356720    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259db7d9_6ddd_4647_96d6_6782765aff7e.slice/crio-80861da7b5102a27aabf7909a5276a47ad76fd585760f8e020d73dd6391d81fe WatchSource:0}: Error finding container 80861da7b5102a27aabf7909a5276a47ad76fd585760f8e020d73dd6391d81fe: Status 404 returned error can't find the container with id 80861da7b5102a27aabf7909a5276a47ad76fd585760f8e020d73dd6391d81fe
Jul 29 16:51:48.358285 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:48.357896    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f45c1b_30e6_4965_8174_24bd4e0f815b.slice/crio-6a4c5c929ffa13aa37e42c1a8834aa67e30cd38505256c0e48506db4b25e4114 WatchSource:0}: Error finding container 6a4c5c929ffa13aa37e42c1a8834aa67e30cd38505256c0e48506db4b25e4114: Status 404 returned error can't find the container with id 6a4c5c929ffa13aa37e42c1a8834aa67e30cd38505256c0e48506db4b25e4114
Jul 29 16:51:48.365254 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.365209    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm"]
Jul 29 16:51:48.368701 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.367314    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zl259"]
Jul 29 16:51:48.368701 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.368480    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn"]
Jul 29 16:51:48.371904 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.371512    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-gg4gt"]
Jul 29 16:51:48.374767 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:48.374544    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240795ba_8cfb_4ab3_b9ac_6d453268441f.slice/crio-8e2845f2c9da610bc261a3d0a74e39b3c6c5119857579516e19057212b617779 WatchSource:0}: Error finding container 8e2845f2c9da610bc261a3d0a74e39b3c6c5119857579516e19057212b617779: Status 404 returned error can't find the container with id 8e2845f2c9da610bc261a3d0a74e39b3c6c5119857579516e19057212b617779
Jul 29 16:51:48.375874 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:51:48.375851    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357c4ea3_b7a7_4f23_8ac7_f06745febc90.slice/crio-0d48fc54826f8023eb9e20b25d4f9ce3f82f7fc46fe861ca45990edb57278e11 WatchSource:0}: Error finding container 0d48fc54826f8023eb9e20b25d4f9ce3f82f7fc46fe861ca45990edb57278e11: Status 404 returned error can't find the container with id 0d48fc54826f8023eb9e20b25d4f9ce3f82f7fc46fe861ca45990edb57278e11
Jul 29 16:51:48.793811 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.793758    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d59pv\" (UID: \"9a232c9c-55d3-4118-90cc-49aae6db92d5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"
Jul 29 16:51:48.794008 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.793910    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 16:51:48.794008 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:48.793980    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert podName:9a232c9c-55d3-4118-90cc-49aae6db92d5 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:49.793964595 +0000 UTC m=+38.283484229 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d59pv" (UID: "9a232c9c-55d3-4118-90cc-49aae6db92d5") : secret "networking-console-plugin-cert" not found
Jul 29 16:51:48.907674 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.907638    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:51:48.907837 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.907711    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:51:48.907837 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.907821    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:51:48.910649 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.910627    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 16:51:48.910764 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.910740    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fgvc\""
Jul 29 16:51:48.910846 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.910798    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5qtk2\""
Jul 29 16:51:48.911423 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:48.911402    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 16:51:49.137819 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.137690    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm" event={"ID":"2c5076ed-8a8a-4866-801a-5b0037ec4d36","Type":"ContainerStarted","Data":"e939985011e0aead25c426a6e2c59a4956274c22727c775a0dc9c9aead7abaea"}
Jul 29 16:51:49.140726 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.140593    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zl259" event={"ID":"240795ba-8cfb-4ab3-b9ac-6d453268441f","Type":"ContainerStarted","Data":"8e2845f2c9da610bc261a3d0a74e39b3c6c5119857579516e19057212b617779"}
Jul 29 16:51:49.142328 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.142241    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zx4jr" event={"ID":"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e","Type":"ContainerStarted","Data":"c9d3005e1f108b8964432d7294a5d85f665c8d9ab1412bd2567b53c6454b15b0"}
Jul 29 16:51:49.152500 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.152335    2570 generic.go:356] "Generic (PLEG): container finished" podID="e356ec32-854a-4b20-985b-e93890a39aa3" containerID="02499b477f47bd64d91d7e833a310f1d9ab1acf37c9dd4d54da6b76293e8b5ef" exitCode=0
Jul 29 16:51:49.152500 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.152441    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxjp" event={"ID":"e356ec32-854a-4b20-985b-e93890a39aa3","Type":"ContainerDied","Data":"02499b477f47bd64d91d7e833a310f1d9ab1acf37c9dd4d54da6b76293e8b5ef"}
Jul 29 16:51:49.155925 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.155847    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn" event={"ID":"a86ddea8-14e1-4ea5-bd16-20e267ef74f1","Type":"ContainerStarted","Data":"6d68459769865b194dbb02af38bc003d39fa9e5c88cba1467ca8e1507c0a18d6"}
Jul 29 16:51:49.160260 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.160110    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tv5bc" event={"ID":"259db7d9-6ddd-4647-96d6-6782765aff7e","Type":"ContainerStarted","Data":"80861da7b5102a27aabf7909a5276a47ad76fd585760f8e020d73dd6391d81fe"}
Jul 29 16:51:49.165119 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.165028    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5mnw6" event={"ID":"50f45c1b-30e6-4965-8174-24bd4e0f815b","Type":"ContainerStarted","Data":"6a4c5c929ffa13aa37e42c1a8834aa67e30cd38505256c0e48506db4b25e4114"}
Jul 29 16:51:49.170676 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.170635    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt" event={"ID":"357c4ea3-b7a7-4f23-8ac7-f06745febc90","Type":"ContainerStarted","Data":"0d48fc54826f8023eb9e20b25d4f9ce3f82f7fc46fe861ca45990edb57278e11"}
Jul 29 16:51:49.197876 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.197835    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:49.198103 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.197893    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:51:49.198103 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.198004    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-stpc7\" (UID: \"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"
Jul 29 16:51:49.198201 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:49.198155    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 16:51:49.198268 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:49.198235    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls podName:ba8efd0d-7b3c-4c24-9443-f7bc8d66b535 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:51.198213992 +0000 UTC m=+39.687733632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls") pod "cluster-samples-operator-6748489656-stpc7" (UID: "ba8efd0d-7b3c-4c24-9443-f7bc8d66b535") : secret "samples-operator-tls" not found
Jul 29 16:51:49.198670 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:49.198651    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 16:51:49.198670 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:49.198672    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8589c7d-4x49t: secret "image-registry-tls" not found
Jul 29 16:51:49.198809 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:49.198716    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls podName:1a5e4084-948e-4a3b-9752-b80d169c5078 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:51.19870163 +0000 UTC m=+39.688221269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls") pod "image-registry-57b8589c7d-4x49t" (UID: "1a5e4084-948e-4a3b-9752-b80d169c5078") : secret "image-registry-tls" not found
Jul 29 16:51:49.198809 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:49.198794    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 16:51:49.198907 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:49.198829    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls podName:8d895e18-c64e-4656-868b-4d5191988051 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:51.198817639 +0000 UTC m=+39.688337275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-787s9" (UID: "8d895e18-c64e-4656-868b-4d5191988051") : secret "cluster-monitoring-operator-tls" not found
Jul 29 16:51:49.299613 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.299546    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:49.299613 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.299608    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:49.299865 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.299695    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:49.300719 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:49.300695    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 16:51:49.300808 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:49.300761    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls podName:b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:51.300742943 +0000 UTC m=+39.790262584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls") pod "dns-default-95x46" (UID: "b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85") : secret "dns-default-metrics-tls" not found
Jul 29 16:51:49.301214 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:49.301195    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle podName:fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:51.301177815 +0000 UTC m=+39.790697464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle") pod "router-default-5df7c8cd4c-mgz6p" (UID: "fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6") : configmap references non-existent config key: service-ca.crt
Jul 29 16:51:49.301634 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:49.301617    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 16:51:49.301706 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:49.301662    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs podName:fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:51.301649688 +0000 UTC m=+39.791169323 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs") pod "router-default-5df7c8cd4c-mgz6p" (UID: "fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6") : secret "router-metrics-certs-default" not found
Jul 29 16:51:49.806050 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:49.805370    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d59pv\" (UID: \"9a232c9c-55d3-4118-90cc-49aae6db92d5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"
Jul 29 16:51:49.806050 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:49.805618    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 16:51:49.806050 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:49.805684    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert podName:9a232c9c-55d3-4118-90cc-49aae6db92d5 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:51.805664748 +0000 UTC m=+40.295184389 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d59pv" (UID: "9a232c9c-55d3-4118-90cc-49aae6db92d5") : secret "networking-console-plugin-cert" not found
Jul 29 16:51:50.182703 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:50.182397    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxjp" event={"ID":"e356ec32-854a-4b20-985b-e93890a39aa3","Type":"ContainerDied","Data":"c4390eb59532f1fb4686fca5f771de74ea1646c7631b4e7926b9a61d86b5af50"}
Jul 29 16:51:50.183456 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:50.183047    2570 generic.go:356] "Generic (PLEG): container finished" podID="e356ec32-854a-4b20-985b-e93890a39aa3" containerID="c4390eb59532f1fb4686fca5f771de74ea1646c7631b4e7926b9a61d86b5af50" exitCode=0
Jul 29 16:51:50.189513 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:50.189389    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5mnw6" event={"ID":"50f45c1b-30e6-4965-8174-24bd4e0f815b","Type":"ContainerStarted","Data":"aff3aa4cae8de378d4b27f8aecd9325174a32770cfe8850abc77fb339817da67"}
Jul 29 16:51:50.233382 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:50.233279    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5mnw6" podStartSLOduration=5.748856504 podStartE2EDuration="7.233256533s" totalImagesPullingTime="1.484400029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:50.232386497 +0000 UTC m=+38.721906154" watchObservedRunningTime="2026-07-29 16:51:50.233256533 +0000 UTC m=+38.722776189"
Jul 29 16:51:51.197473 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:51.197370    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxjp" event={"ID":"e356ec32-854a-4b20-985b-e93890a39aa3","Type":"ContainerStarted","Data":"faf8229b6c149824b0d63f030be971cf76d4ec0157a1516f35d8ef7e17a06d63"}
Jul 29 16:51:51.228594 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:51.228541    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-stpc7\" (UID: \"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"
Jul 29 16:51:51.228834 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:51.228813    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:51.228925 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:51.228856    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:51:51.229036 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:51.228686    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 16:51:51.229106 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:51.229094    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls podName:ba8efd0d-7b3c-4c24-9443-f7bc8d66b535 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:55.229075194 +0000 UTC m=+43.718594843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls") pod "cluster-samples-operator-6748489656-stpc7" (UID: "ba8efd0d-7b3c-4c24-9443-f7bc8d66b535") : secret "samples-operator-tls" not found
Jul 29 16:51:51.229525 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:51.229507    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 16:51:51.229525 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:51.229524    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8589c7d-4x49t: secret "image-registry-tls" not found
Jul 29 16:51:51.229713 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:51.229569    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls podName:1a5e4084-948e-4a3b-9752-b80d169c5078 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:55.229555209 +0000 UTC m=+43.719074856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls") pod "image-registry-57b8589c7d-4x49t" (UID: "1a5e4084-948e-4a3b-9752-b80d169c5078") : secret "image-registry-tls" not found
Jul 29 16:51:51.229713 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:51.229020    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 16:51:51.229713 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:51.229609    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls podName:8d895e18-c64e-4656-868b-4d5191988051 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:55.229600158 +0000 UTC m=+43.719119799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-787s9" (UID: "8d895e18-c64e-4656-868b-4d5191988051") : secret "cluster-monitoring-operator-tls" not found
Jul 29 16:51:51.231116 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:51.231065    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lsxjp" podStartSLOduration=8.662962689 podStartE2EDuration="39.23104821s" totalImagesPullingTime="30.568085521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:51.230747756 +0000 UTC m=+39.720267413" watchObservedRunningTime="2026-07-29 16:51:51.23104821 +0000 UTC m=+39.720567867"
Jul 29 16:51:51.331080 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:51.329918    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:51.331080 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:51.330037    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:51.331080 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:51.330074    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle podName:fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:55.330052409 +0000 UTC m=+43.819572056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle") pod "router-default-5df7c8cd4c-mgz6p" (UID: "fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6") : configmap references non-existent config key: service-ca.crt
Jul 29 16:51:51.331080 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:51.330130    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 16:51:51.331080 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:51.330186    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs podName:fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:55.330156631 +0000 UTC m=+43.819676281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs") pod "router-default-5df7c8cd4c-mgz6p" (UID: "fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6") : secret "router-metrics-certs-default" not found
Jul 29 16:51:51.331080 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:51.330349    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:51.331080 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:51.330445    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 16:51:51.331080 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:51.330483    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls podName:b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:55.33046887 +0000 UTC m=+43.819988518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls") pod "dns-default-95x46" (UID: "b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85") : secret "dns-default-metrics-tls" not found
Jul 29 16:51:51.837443 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:51.837350    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d59pv\" (UID: \"9a232c9c-55d3-4118-90cc-49aae6db92d5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"
Jul 29 16:51:51.837653 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:51.837528    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 16:51:51.837653 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:51.837613    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert podName:9a232c9c-55d3-4118-90cc-49aae6db92d5 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:55.837591253 +0000 UTC m=+44.327110888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d59pv" (UID: "9a232c9c-55d3-4118-90cc-49aae6db92d5") : secret "networking-console-plugin-cert" not found
Jul 29 16:51:55.267767 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:55.267724    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-stpc7\" (UID: \"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"
Jul 29 16:51:55.268293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:55.267840    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:51:55.268293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:55.267875    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:51:55.268293 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:55.267894    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 16:51:55.268293 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:55.267974    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls podName:ba8efd0d-7b3c-4c24-9443-f7bc8d66b535 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:03.267953015 +0000 UTC m=+51.757472672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls") pod "cluster-samples-operator-6748489656-stpc7" (UID: "ba8efd0d-7b3c-4c24-9443-f7bc8d66b535") : secret "samples-operator-tls" not found
Jul 29 16:51:55.268293 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:55.267981    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 16:51:55.268293 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:55.268024    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls podName:8d895e18-c64e-4656-868b-4d5191988051 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:03.268014429 +0000 UTC m=+51.757534063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-787s9" (UID: "8d895e18-c64e-4656-868b-4d5191988051") : secret "cluster-monitoring-operator-tls" not found
Jul 29 16:51:55.268293 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:55.268047    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 16:51:55.268293 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:55.268075    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8589c7d-4x49t: secret "image-registry-tls" not found
Jul 29 16:51:55.268293 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:55.268200    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls podName:1a5e4084-948e-4a3b-9752-b80d169c5078 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:03.268178442 +0000 UTC m=+51.757698093 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls") pod "image-registry-57b8589c7d-4x49t" (UID: "1a5e4084-948e-4a3b-9752-b80d169c5078") : secret "image-registry-tls" not found
Jul 29 16:51:55.368864 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:55.368813    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:51:55.369043 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:55.368891    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:55.369043 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:55.368941    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:51:55.369043 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:55.368941    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 16:51:55.369043 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:55.369013    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls podName:b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:03.368993075 +0000 UTC m=+51.858512708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls") pod "dns-default-95x46" (UID: "b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85") : secret "dns-default-metrics-tls" not found
Jul 29 16:51:55.369043 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:55.369034    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle podName:fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:03.369022792 +0000 UTC m=+51.858542431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle") pod "router-default-5df7c8cd4c-mgz6p" (UID: "fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6") : configmap references non-existent config key: service-ca.crt
Jul 29 16:51:55.369357 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:55.369071    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 16:51:55.369357 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:55.369126    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs podName:fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:03.369109031 +0000 UTC m=+51.858628688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs") pod "router-default-5df7c8cd4c-mgz6p" (UID: "fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6") : secret "router-metrics-certs-default" not found
Jul 29 16:51:55.874041 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:55.873999    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d59pv\" (UID: \"9a232c9c-55d3-4118-90cc-49aae6db92d5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"
Jul 29 16:51:55.874230 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:55.874118    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 16:51:55.874230 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:55.874200    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert podName:9a232c9c-55d3-4118-90cc-49aae6db92d5 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:03.87418465 +0000 UTC m=+52.363704284 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d59pv" (UID: "9a232c9c-55d3-4118-90cc-49aae6db92d5") : secret "networking-console-plugin-cert" not found
Jul 29 16:51:57.218838 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.218800    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm" event={"ID":"2c5076ed-8a8a-4866-801a-5b0037ec4d36","Type":"ContainerStarted","Data":"7334e741781ce8b9b0c6f8090b475692503fe9a8ddfa733484309d897c2e6d22"}
Jul 29 16:51:57.221080 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.221043    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zl259" event={"ID":"240795ba-8cfb-4ab3-b9ac-6d453268441f","Type":"ContainerStarted","Data":"f74d71764d9e0d37a1b9ae5e8c89e09121d4b39efce9cdbb6c9ca45193d99b41"}
Jul 29 16:51:57.222869 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.222837    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zx4jr" event={"ID":"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e","Type":"ContainerStarted","Data":"02481bcb5d6bc4038c352af4cc3bc816166b8b55092664f028649dcaf03470bb"}
Jul 29 16:51:57.224726 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.224697    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn" event={"ID":"a86ddea8-14e1-4ea5-bd16-20e267ef74f1","Type":"ContainerStarted","Data":"2674d464dccff6e54fa97cab2e2b3b2e0011e912259cb31bbf3f13b0aec7c874"}
Jul 29 16:51:57.226616 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.226584    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tv5bc" event={"ID":"259db7d9-6ddd-4647-96d6-6782765aff7e","Type":"ContainerStarted","Data":"57f2d85e2193344008753d8d1631e3a4a594032c6ae20a6636d9351e8bcf81f2"}
Jul 29 16:51:57.228445 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.228387    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/0.log"
Jul 29 16:51:57.228571 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.228466    2570 generic.go:356] "Generic (PLEG): container finished" podID="357c4ea3-b7a7-4f23-8ac7-f06745febc90" containerID="3a1f04cacaceb9c189288605e6b5c60b4ebe3e2a80d207796bb7a8c473b16c02" exitCode=255
Jul 29 16:51:57.228571 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.228505    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt" event={"ID":"357c4ea3-b7a7-4f23-8ac7-f06745febc90","Type":"ContainerDied","Data":"3a1f04cacaceb9c189288605e6b5c60b4ebe3e2a80d207796bb7a8c473b16c02"}
Jul 29 16:51:57.228765 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.228747    2570 scope.go:117] "RemoveContainer" containerID="3a1f04cacaceb9c189288605e6b5c60b4ebe3e2a80d207796bb7a8c473b16c02"
Jul 29 16:51:57.237980 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.237578    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm" podStartSLOduration=6.471690768 podStartE2EDuration="14.237556157s" totalImagesPullingTime="7.765865389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:57.237140102 +0000 UTC m=+45.726659759" watchObservedRunningTime="2026-07-29 16:51:57.237556157 +0000 UTC m=+45.727075815"
Jul 29 16:51:57.278961 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.278852    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-zx4jr" podStartSLOduration=6.494814191 podStartE2EDuration="14.278827293s" totalImagesPullingTime="7.784013102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:57.255407578 +0000 UTC m=+45.744927235" watchObservedRunningTime="2026-07-29 16:51:57.278827293 +0000 UTC m=+45.768346949"
Jul 29 16:51:57.311437 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.311367    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tv5bc" podStartSLOduration=6.518924638 podStartE2EDuration="14.311341242s" totalImagesPullingTime="7.792416604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:57.309645717 +0000 UTC m=+45.799165374" watchObservedRunningTime="2026-07-29 16:51:57.311341242 +0000 UTC m=+45.800860941"
Jul 29 16:51:57.342939 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.342878    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zl259" podStartSLOduration=2.579768489 podStartE2EDuration="10.342855789s" totalImagesPullingTime="7.7630873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:57.341990748 +0000 UTC m=+45.831510405" watchObservedRunningTime="2026-07-29 16:51:57.342855789 +0000 UTC m=+45.832375450"
Jul 29 16:51:57.389350 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.389282    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn" podStartSLOduration=6.62411812 podStartE2EDuration="14.389254082s" totalImagesPullingTime="7.765135962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:57.389198336 +0000 UTC m=+45.878717993" watchObservedRunningTime="2026-07-29 16:51:57.389254082 +0000 UTC m=+45.878773738"
Jul 29 16:51:57.827536 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.827485    2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:57.827742 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:57.827597    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:51:58.234838 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.234752    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fwrgv"]
Jul 29 16:51:58.236477 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.236449    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 16:51:58.237039 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.237016    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/0.log"
Jul 29 16:51:58.237190 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.237061    2570 generic.go:356] "Generic (PLEG): container finished" podID="357c4ea3-b7a7-4f23-8ac7-f06745febc90" containerID="3d04ced75c5f2527e46d8bd3a4c45678cee4e1eb6a27cb0670c9b6fef7652697" exitCode=255
Jul 29 16:51:58.238043 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.238020    2570 scope.go:117] "RemoveContainer" containerID="3d04ced75c5f2527e46d8bd3a4c45678cee4e1eb6a27cb0670c9b6fef7652697"
Jul 29 16:51:58.238208 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:58.238184    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-gg4gt_openshift-console-operator(357c4ea3-b7a7-4f23-8ac7-f06745febc90)\"" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt" podUID="357c4ea3-b7a7-4f23-8ac7-f06745febc90"
Jul 29 16:51:58.259959 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.259927    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fwrgv"]
Jul 29 16:51:58.260123 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.259981    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt" event={"ID":"357c4ea3-b7a7-4f23-8ac7-f06745febc90","Type":"ContainerDied","Data":"3d04ced75c5f2527e46d8bd3a4c45678cee4e1eb6a27cb0670c9b6fef7652697"}
Jul 29 16:51:58.260123 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.260018    2570 scope.go:117] "RemoveContainer" containerID="3a1f04cacaceb9c189288605e6b5c60b4ebe3e2a80d207796bb7a8c473b16c02"
Jul 29 16:51:58.260278 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.260129    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.262513 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.262488    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 16:51:58.262834 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.262814    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gbp95\""
Jul 29 16:51:58.262942 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.262921    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 16:51:58.305101 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.305069    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdqr4\" (UniqueName: \"kubernetes.io/projected/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-api-access-fdqr4\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.305375 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.305218    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84e4fe40-6165-40b3-9f20-3d3003e57759-data-volume\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.305375 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.305246    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.305375 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.305300    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.306144 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.306111    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84e4fe40-6165-40b3-9f20-3d3003e57759-crio-socket\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.407568 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.407522    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84e4fe40-6165-40b3-9f20-3d3003e57759-crio-socket\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.407768 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.407586    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdqr4\" (UniqueName: \"kubernetes.io/projected/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-api-access-fdqr4\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.407768 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.407626    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84e4fe40-6165-40b3-9f20-3d3003e57759-data-volume\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.407768 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.407643    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.407768 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.407669    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.407768 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.407722    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84e4fe40-6165-40b3-9f20-3d3003e57759-crio-socket\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.408022 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:58.407801    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 16:51:58.408022 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:58.407859    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 16:51:58.408022 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:58.407895    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls podName:84e4fe40-6165-40b3-9f20-3d3003e57759 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:58.907870526 +0000 UTC m=+47.397390162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fwrgv" (UID: "84e4fe40-6165-40b3-9f20-3d3003e57759") : secret "insights-runtime-extractor-tls" not found
Jul 29 16:51:58.408022 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:58.407916    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm podName:84e4fe40-6165-40b3-9f20-3d3003e57759 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:58.907905803 +0000 UTC m=+47.397425443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fwrgv" (UID: "84e4fe40-6165-40b3-9f20-3d3003e57759") : configmap "kube-rbac-proxy" not found
Jul 29 16:51:58.408022 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.407994    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84e4fe40-6165-40b3-9f20-3d3003e57759-data-volume\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.422109 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.422078    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdqr4\" (UniqueName: \"kubernetes.io/projected/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-api-access-fdqr4\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.911818 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.911780    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.911818 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:58.911822    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:58.912048 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:58.911929    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 16:51:58.912048 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:58.911939    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 16:51:58.912048 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:58.911983    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm podName:84e4fe40-6165-40b3-9f20-3d3003e57759 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:59.911970433 +0000 UTC m=+48.401490067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fwrgv" (UID: "84e4fe40-6165-40b3-9f20-3d3003e57759") : configmap "kube-rbac-proxy" not found
Jul 29 16:51:58.912048 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:58.912011    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls podName:84e4fe40-6165-40b3-9f20-3d3003e57759 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:59.911997182 +0000 UTC m=+48.401516816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fwrgv" (UID: "84e4fe40-6165-40b3-9f20-3d3003e57759") : secret "insights-runtime-extractor-tls" not found
Jul 29 16:51:59.246219 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:59.246128    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 16:51:59.246582 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:59.246511    2570 scope.go:117] "RemoveContainer" containerID="3d04ced75c5f2527e46d8bd3a4c45678cee4e1eb6a27cb0670c9b6fef7652697"
Jul 29 16:51:59.246681 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:59.246660    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-gg4gt_openshift-console-operator(357c4ea3-b7a7-4f23-8ac7-f06745febc90)\"" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt" podUID="357c4ea3-b7a7-4f23-8ac7-f06745febc90"
Jul 29 16:51:59.631751 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:59.631723    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dq655_8ed06670-f24d-4521-815a-0d27741edac3/dns-node-resolver/0.log"
Jul 29 16:51:59.921098 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:59.920998    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:59.921098 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:51:59.921047    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:51:59.921334 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:59.921147    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 16:51:59.921334 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:59.921201    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 16:51:59.921334 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:59.921239    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls podName:84e4fe40-6165-40b3-9f20-3d3003e57759 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:01.921215394 +0000 UTC m=+50.410735045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fwrgv" (UID: "84e4fe40-6165-40b3-9f20-3d3003e57759") : secret "insights-runtime-extractor-tls" not found
Jul 29 16:51:59.921334 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:51:59.921260    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm podName:84e4fe40-6165-40b3-9f20-3d3003e57759 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:01.921250889 +0000 UTC m=+50.410770525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fwrgv" (UID: "84e4fe40-6165-40b3-9f20-3d3003e57759") : configmap "kube-rbac-proxy" not found
Jul 29 16:52:00.249935 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:00.249863    2570 scope.go:117] "RemoveContainer" containerID="3d04ced75c5f2527e46d8bd3a4c45678cee4e1eb6a27cb0670c9b6fef7652697"
Jul 29 16:52:00.250307 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:00.250014    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-gg4gt_openshift-console-operator(357c4ea3-b7a7-4f23-8ac7-f06745febc90)\"" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt" podUID="357c4ea3-b7a7-4f23-8ac7-f06745febc90"
Jul 29 16:52:00.636577 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:00.636545    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6qp9p_e22624ca-991d-4e4d-8db7-02119caf746d/node-ca/0.log"
Jul 29 16:52:01.634756 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:01.634726    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zl259_240795ba-8cfb-4ab3-b9ac-6d453268441f/serve-healthcheck-canary/0.log"
Jul 29 16:52:01.942052 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:01.941961    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:52:01.942052 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:01.942001    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:52:01.942293 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:01.942107    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 16:52:01.942293 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:01.942116    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 16:52:01.942293 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:01.942194    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls podName:84e4fe40-6165-40b3-9f20-3d3003e57759 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:05.942173047 +0000 UTC m=+54.431692696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fwrgv" (UID: "84e4fe40-6165-40b3-9f20-3d3003e57759") : secret "insights-runtime-extractor-tls" not found
Jul 29 16:52:01.942293 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:01.942228    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm podName:84e4fe40-6165-40b3-9f20-3d3003e57759 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:05.942207019 +0000 UTC m=+54.431726657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fwrgv" (UID: "84e4fe40-6165-40b3-9f20-3d3003e57759") : configmap "kube-rbac-proxy" not found
Jul 29 16:52:03.355780 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:03.355730    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-stpc7\" (UID: \"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"
Jul 29 16:52:03.356216 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:03.355840    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:52:03.356216 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:03.355864    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:52:03.356216 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:03.355890    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 16:52:03.356216 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:03.355960    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls podName:ba8efd0d-7b3c-4c24-9443-f7bc8d66b535 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:19.355943133 +0000 UTC m=+67.845462767 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls") pod "cluster-samples-operator-6748489656-stpc7" (UID: "ba8efd0d-7b3c-4c24-9443-f7bc8d66b535") : secret "samples-operator-tls" not found
Jul 29 16:52:03.356216 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:03.355977    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 16:52:03.356216 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:03.356017    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 16:52:03.356216 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:03.356037    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8589c7d-4x49t: secret "image-registry-tls" not found
Jul 29 16:52:03.356216 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:03.356029    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls podName:8d895e18-c64e-4656-868b-4d5191988051 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:19.356015098 +0000 UTC m=+67.845534733 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-787s9" (UID: "8d895e18-c64e-4656-868b-4d5191988051") : secret "cluster-monitoring-operator-tls" not found
Jul 29 16:52:03.356216 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:03.356120    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls podName:1a5e4084-948e-4a3b-9752-b80d169c5078 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:19.356110643 +0000 UTC m=+67.845630277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls") pod "image-registry-57b8589c7d-4x49t" (UID: "1a5e4084-948e-4a3b-9752-b80d169c5078") : secret "image-registry-tls" not found
Jul 29 16:52:03.457285 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:03.457244    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:52:03.457285 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:03.457292    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:52:03.457494 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:03.457383    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:52:03.457494 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:03.457412    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle podName:fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:19.457399016 +0000 UTC m=+67.946918650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle") pod "router-default-5df7c8cd4c-mgz6p" (UID: "fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6") : configmap references non-existent config key: service-ca.crt
Jul 29 16:52:03.457494 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:03.457452    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 16:52:03.457593 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:03.457500    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs podName:fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:19.45748524 +0000 UTC m=+67.947004890 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs") pod "router-default-5df7c8cd4c-mgz6p" (UID: "fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6") : secret "router-metrics-certs-default" not found
Jul 29 16:52:03.457593 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:03.457501    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 16:52:03.457593 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:03.457553    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls podName:b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:19.457541658 +0000 UTC m=+67.947061292 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls") pod "dns-default-95x46" (UID: "b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85") : secret "dns-default-metrics-tls" not found
Jul 29 16:52:03.961295 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:03.961255    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d59pv\" (UID: \"9a232c9c-55d3-4118-90cc-49aae6db92d5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"
Jul 29 16:52:03.961484 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:03.961413    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 16:52:03.961484 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:03.961480    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert podName:9a232c9c-55d3-4118-90cc-49aae6db92d5 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:19.961464925 +0000 UTC m=+68.450984560 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d59pv" (UID: "9a232c9c-55d3-4118-90cc-49aae6db92d5") : secret "networking-console-plugin-cert" not found
Jul 29 16:52:05.977440 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:05.977400    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:52:05.977440 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:05.977437    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:52:05.977929 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:05.977581    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 16:52:05.977929 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:05.977659    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls podName:84e4fe40-6165-40b3-9f20-3d3003e57759 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:13.977640312 +0000 UTC m=+62.467159948 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fwrgv" (UID: "84e4fe40-6165-40b3-9f20-3d3003e57759") : secret "insights-runtime-extractor-tls" not found
Jul 29 16:52:05.978020 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:05.977982    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84e4fe40-6165-40b3-9f20-3d3003e57759-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:52:07.827518 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:07.827466    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:52:07.827518 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:07.827536    2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:52:07.828035 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:07.827866    2570 scope.go:117] "RemoveContainer" containerID="3d04ced75c5f2527e46d8bd3a4c45678cee4e1eb6a27cb0670c9b6fef7652697"
Jul 29 16:52:08.278348 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:08.278319    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 16:52:08.278519 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:08.278366    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt" event={"ID":"357c4ea3-b7a7-4f23-8ac7-f06745febc90","Type":"ContainerStarted","Data":"03abee8f1de975fc6cb7f89e81697e80c1c0a955c7b63f331776c6aff9ee2d91"}
Jul 29 16:52:08.278707 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:08.278679    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:52:08.299331 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:08.299262    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt" podStartSLOduration=17.537983761 podStartE2EDuration="25.299229531s" totalImagesPullingTime="7.76124577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:08.296715934 +0000 UTC m=+56.786235593" watchObservedRunningTime="2026-07-29 16:52:08.299229531 +0000 UTC m=+56.788749188"
Jul 29 16:52:08.337632 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:08.337593    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-gg4gt"
Jul 29 16:52:11.116380 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:11.116353    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qfpbq"
Jul 29 16:52:14.046969 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:14.046921    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:52:14.049353 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:14.049329    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84e4fe40-6165-40b3-9f20-3d3003e57759-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fwrgv\" (UID: \"84e4fe40-6165-40b3-9f20-3d3003e57759\") " pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:52:14.173361 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:14.173325    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gbp95\""
Jul 29 16:52:14.181208 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:14.181185    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fwrgv"
Jul 29 16:52:14.313896 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:14.313825    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fwrgv"]
Jul 29 16:52:14.317179 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:52:14.317136    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e4fe40_6165_40b3_9f20_3d3003e57759.slice/crio-ee228a9d3c37aef7a8fa39dcaaea1adba12154c7b15c091ba0f940c63dcca87f WatchSource:0}: Error finding container ee228a9d3c37aef7a8fa39dcaaea1adba12154c7b15c091ba0f940c63dcca87f: Status 404 returned error can't find the container with id ee228a9d3c37aef7a8fa39dcaaea1adba12154c7b15c091ba0f940c63dcca87f
Jul 29 16:52:15.304433 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:15.304383    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fwrgv" event={"ID":"84e4fe40-6165-40b3-9f20-3d3003e57759","Type":"ContainerStarted","Data":"ee228a9d3c37aef7a8fa39dcaaea1adba12154c7b15c091ba0f940c63dcca87f"}
Jul 29 16:52:17.686607 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:17.686555    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:52:17.687059 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:17.686658    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qd5\" (UniqueName: \"kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5\") pod \"network-check-target-zb222\" (UID: \"335e89f7-203c-4d41-8d24-b2fde7765514\") " pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:52:17.687059 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:17.686701    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs\") pod \"network-metrics-daemon-pmqs8\" (UID: \"438c3501-4812-489f-ae51-a25e01b9aacc\") " pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:52:17.689702 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:17.689667    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97qd5\" (UniqueName: \"kubernetes.io/projected/335e89f7-203c-4d41-8d24-b2fde7765514-kube-api-access-97qd5\") pod \"network-check-target-zb222\" (UID: \"335e89f7-203c-4d41-8d24-b2fde7765514\") " pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:52:17.690348 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:17.690323    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 16:52:17.690463 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:17.690413    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 16:52:17.699834 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:17.699808    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438c3501-4812-489f-ae51-a25e01b9aacc-metrics-certs\") pod \"network-metrics-daemon-pmqs8\" (UID: \"438c3501-4812-489f-ae51-a25e01b9aacc\") " pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:52:17.699991 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:17.699868    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bd992f-213e-46bd-87a6-b06f7895719d-original-pull-secret\") pod \"global-pull-secret-syncer-dxzh5\" (UID: \"b1bd992f-213e-46bd-87a6-b06f7895719d\") " pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:52:17.720925 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:17.720883    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxzh5"
Jul 29 16:52:17.730598 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:17.730563    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5qtk2\""
Jul 29 16:52:17.736039 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:17.736011    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fgvc\""
Jul 29 16:52:17.738202 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:17.738157    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:52:17.744322 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:17.744301    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pmqs8"
Jul 29 16:52:19.335133 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.335102    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zb222"]
Jul 29 16:52:19.350507 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.350473    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pmqs8"]
Jul 29 16:52:19.369073 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.369041    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dxzh5"]
Jul 29 16:52:19.400582 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:52:19.400522    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335e89f7_203c_4d41_8d24_b2fde7765514.slice/crio-bac08f130a2476475bcdd767204093c80334213c5eaeeb5efec378c869e46654 WatchSource:0}: Error finding container bac08f130a2476475bcdd767204093c80334213c5eaeeb5efec378c869e46654: Status 404 returned error can't find the container with id bac08f130a2476475bcdd767204093c80334213c5eaeeb5efec378c869e46654
Jul 29 16:52:19.401728 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:52:19.401487    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438c3501_4812_489f_ae51_a25e01b9aacc.slice/crio-652b4739eebb39decb73a6dd5610fd1b8cae22483898fd896c27881935bd036c WatchSource:0}: Error finding container 652b4739eebb39decb73a6dd5610fd1b8cae22483898fd896c27881935bd036c: Status 404 returned error can't find the container with id 652b4739eebb39decb73a6dd5610fd1b8cae22483898fd896c27881935bd036c
Jul 29 16:52:19.404565 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.404266    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:52:19.404565 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.404383    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-stpc7\" (UID: \"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"
Jul 29 16:52:19.404565 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:52:19.404461    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bd992f_213e_46bd_87a6_b06f7895719d.slice/crio-25835a6b247182ff28a790e94ae8c90a67efd153c193dcd815d9f7112ccfcb20 WatchSource:0}: Error finding container 25835a6b247182ff28a790e94ae8c90a67efd153c193dcd815d9f7112ccfcb20: Status 404 returned error can't find the container with id 25835a6b247182ff28a790e94ae8c90a67efd153c193dcd815d9f7112ccfcb20
Jul 29 16:52:19.404565 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.404471    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:52:19.407371 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.407349    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls\") pod \"image-registry-57b8589c7d-4x49t\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") " pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:52:19.407484 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.407428    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d895e18-c64e-4656-868b-4d5191988051-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-787s9\" (UID: \"8d895e18-c64e-4656-868b-4d5191988051\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:52:19.407541 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.407522    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba8efd0d-7b3c-4c24-9443-f7bc8d66b535-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-stpc7\" (UID: \"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"
Jul 29 16:52:19.505122 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.505085    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:52:19.505218 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.505186    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:52:19.505318 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.505301    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:52:19.505852 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.505826    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-service-ca-bundle\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:52:19.507600 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.507579    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85-metrics-tls\") pod \"dns-default-95x46\" (UID: \"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85\") " pod="openshift-dns/dns-default-95x46"
Jul 29 16:52:19.507712 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.507671    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6-metrics-certs\") pod \"router-default-5df7c8cd4c-mgz6p\" (UID: \"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6\") " pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:52:19.597865 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.597840    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-78qw7\""
Jul 29 16:52:19.605734 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.605707    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:52:19.621599 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.621576    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cwhbz\""
Jul 29 16:52:19.629852 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.629837    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"
Jul 29 16:52:19.646951 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.646921    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-v8hks\""
Jul 29 16:52:19.653934 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.653910    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2hb68\""
Jul 29 16:52:19.654935 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.654917    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"
Jul 29 16:52:19.662153 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.662129    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-95x46"
Jul 29 16:52:19.675805 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.675549    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jhsnv\""
Jul 29 16:52:19.682921 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.682849    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:52:19.775984 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.775889    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57b8589c7d-4x49t"]
Jul 29 16:52:19.777862 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:52:19.777771    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a5e4084_948e_4a3b_9752_b80d169c5078.slice/crio-0efc1fabc7663345ebaa2bb4c95d5501fda1b811671e1fcdb176d1a189561e9d WatchSource:0}: Error finding container 0efc1fabc7663345ebaa2bb4c95d5501fda1b811671e1fcdb176d1a189561e9d: Status 404 returned error can't find the container with id 0efc1fabc7663345ebaa2bb4c95d5501fda1b811671e1fcdb176d1a189561e9d
Jul 29 16:52:19.834621 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.834533    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7"]
Jul 29 16:52:19.865258 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.865217    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9"]
Jul 29 16:52:19.917147 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.917121    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-95x46"]
Jul 29 16:52:19.917558 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:52:19.917536    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e4b6d0_93b5_4b86_9249_9ba2f27c8c85.slice/crio-528ac3592a5b778066a8f81578887358f5270a6651de1aad4646af2e78b2dfab WatchSource:0}: Error finding container 528ac3592a5b778066a8f81578887358f5270a6651de1aad4646af2e78b2dfab: Status 404 returned error can't find the container with id 528ac3592a5b778066a8f81578887358f5270a6651de1aad4646af2e78b2dfab
Jul 29 16:52:19.945198 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:19.945144    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5df7c8cd4c-mgz6p"]
Jul 29 16:52:19.948868 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:52:19.948777    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3c57cf_294f_4d44_bb5d_8edfb8a0b7d6.slice/crio-3cb258c5db2b7b987747f4d5dfc536881b28fb0a38dd3653dfc21a45bae091fb WatchSource:0}: Error finding container 3cb258c5db2b7b987747f4d5dfc536881b28fb0a38dd3653dfc21a45bae091fb: Status 404 returned error can't find the container with id 3cb258c5db2b7b987747f4d5dfc536881b28fb0a38dd3653dfc21a45bae091fb
Jul 29 16:52:20.011446 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.011415    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d59pv\" (UID: \"9a232c9c-55d3-4118-90cc-49aae6db92d5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"
Jul 29 16:52:20.014183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.014135    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a232c9c-55d3-4118-90cc-49aae6db92d5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d59pv\" (UID: \"9a232c9c-55d3-4118-90cc-49aae6db92d5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"
Jul 29 16:52:20.227864 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.227742    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-njmsx\""
Jul 29 16:52:20.236123 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.235886    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"
Jul 29 16:52:20.329212 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.329154    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p" event={"ID":"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6","Type":"ContainerStarted","Data":"de11b610d520883bd5a44e24ad95328f00f64c00dca21f2dfa419d5289eeab51"}
Jul 29 16:52:20.329378 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.329239    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p" event={"ID":"fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6","Type":"ContainerStarted","Data":"3cb258c5db2b7b987747f4d5dfc536881b28fb0a38dd3653dfc21a45bae091fb"}
Jul 29 16:52:20.334139 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.332517    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b8589c7d-4x49t" event={"ID":"1a5e4084-948e-4a3b-9752-b80d169c5078","Type":"ContainerStarted","Data":"6eafa9f25e1abba92d5a6cc01a55cf64d1bd99b2921625e0d7cd8fa2a873c280"}
Jul 29 16:52:20.334139 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.332574    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b8589c7d-4x49t" event={"ID":"1a5e4084-948e-4a3b-9752-b80d169c5078","Type":"ContainerStarted","Data":"0efc1fabc7663345ebaa2bb4c95d5501fda1b811671e1fcdb176d1a189561e9d"}
Jul 29 16:52:20.334139 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.333154    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:52:20.336379 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.336280    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9" event={"ID":"8d895e18-c64e-4656-868b-4d5191988051","Type":"ContainerStarted","Data":"c92f76b7e2bccdb7725d7a37b15c8244be7e22b56db62a4505666cd1a67170d8"}
Jul 29 16:52:20.338276 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.338201    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pmqs8" event={"ID":"438c3501-4812-489f-ae51-a25e01b9aacc","Type":"ContainerStarted","Data":"652b4739eebb39decb73a6dd5610fd1b8cae22483898fd896c27881935bd036c"}
Jul 29 16:52:20.340421 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.340326    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fwrgv" event={"ID":"84e4fe40-6165-40b3-9f20-3d3003e57759","Type":"ContainerStarted","Data":"8b5542664803e91eb44319515fd5ccdc61a1c776ec00dd3067281da238bd89f7"}
Jul 29 16:52:20.342085 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.342032    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-95x46" event={"ID":"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85","Type":"ContainerStarted","Data":"528ac3592a5b778066a8f81578887358f5270a6651de1aad4646af2e78b2dfab"}
Jul 29 16:52:20.344322 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.344251    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7" event={"ID":"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535","Type":"ContainerStarted","Data":"e39486ad39525049168c3dbfba7ab58e34c0772e913464f970b48e9020eefde5"}
Jul 29 16:52:20.346828 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.346267    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zb222" event={"ID":"335e89f7-203c-4d41-8d24-b2fde7765514","Type":"ContainerStarted","Data":"4dfe52e9cfd9df9524da55d0577f072d9f39adcddb48eb9fda6f703fe706eb78"}
Jul 29 16:52:20.346828 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.346315    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zb222" event={"ID":"335e89f7-203c-4d41-8d24-b2fde7765514","Type":"ContainerStarted","Data":"bac08f130a2476475bcdd767204093c80334213c5eaeeb5efec378c869e46654"}
Jul 29 16:52:20.346828 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.346549    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:52:20.348279 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.348253    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dxzh5" event={"ID":"b1bd992f-213e-46bd-87a6-b06f7895719d","Type":"ContainerStarted","Data":"25835a6b247182ff28a790e94ae8c90a67efd153c193dcd815d9f7112ccfcb20"}
Jul 29 16:52:20.351105 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.351055    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p" podStartSLOduration=37.351034859 podStartE2EDuration="37.351034859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:20.350182128 +0000 UTC m=+68.839701785" watchObservedRunningTime="2026-07-29 16:52:20.351034859 +0000 UTC m=+68.840554515"
Jul 29 16:52:20.388511 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.388417    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zb222" podStartSLOduration=68.388395658 podStartE2EDuration="1m8.388395658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:20.367137654 +0000 UTC m=+68.856657311" watchObservedRunningTime="2026-07-29 16:52:20.388395658 +0000 UTC m=+68.877915315"
Jul 29 16:52:20.388835 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.388781    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57b8589c7d-4x49t" podStartSLOduration=68.388766851 podStartE2EDuration="1m8.388766851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:20.387521534 +0000 UTC m=+68.877041210" watchObservedRunningTime="2026-07-29 16:52:20.388766851 +0000 UTC m=+68.878286508"
Jul 29 16:52:20.445855 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.445770    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-d59pv"]
Jul 29 16:52:20.683481 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.683361    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:52:20.686234 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:20.686187    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:52:20.742597 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:52:20.742560    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a232c9c_55d3_4118_90cc_49aae6db92d5.slice/crio-3c4215c0ef4de46e4e6d31383dbaef17448ae20a9f6382846e2737a23183712b WatchSource:0}: Error finding container 3c4215c0ef4de46e4e6d31383dbaef17448ae20a9f6382846e2737a23183712b: Status 404 returned error can't find the container with id 3c4215c0ef4de46e4e6d31383dbaef17448ae20a9f6382846e2737a23183712b
Jul 29 16:52:21.358979 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:21.358415    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pmqs8" event={"ID":"438c3501-4812-489f-ae51-a25e01b9aacc","Type":"ContainerStarted","Data":"19ab6f25dd58d4897804f112637b53344e44bad1fb038922b150f7a8f275a10c"}
Jul 29 16:52:21.362656 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:21.362623    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv" event={"ID":"9a232c9c-55d3-4118-90cc-49aae6db92d5","Type":"ContainerStarted","Data":"3c4215c0ef4de46e4e6d31383dbaef17448ae20a9f6382846e2737a23183712b"}
Jul 29 16:52:21.362854 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:21.362682    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:52:21.364100 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:21.364075    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5df7c8cd4c-mgz6p"
Jul 29 16:52:22.091044 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.089715    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"]
Jul 29 16:52:22.110729 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.110692    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"
Jul 29 16:52:22.114713 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.114684    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 16:52:22.115090 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.115068    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 16:52:22.117005 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.116979    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"]
Jul 29 16:52:22.117964 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.117537    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 16:52:22.117964 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.117797    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 29 16:52:22.165739 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.165109    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-45wpw"]
Jul 29 16:52:22.189568 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.188522    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-45wpw"]
Jul 29 16:52:22.189568 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.188647    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-45wpw"
Jul 29 16:52:22.193275 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.193008    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 16:52:22.193275 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.193070    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-842bk\""
Jul 29 16:52:22.193275 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.193265    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 16:52:22.240242 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.240204    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjl2n\" (UniqueName: \"kubernetes.io/projected/62942548-83af-4767-8de6-1c2b4085ef50-kube-api-access-hjl2n\") pod \"klusterlet-addon-workmgr-d9788f-lv2tw\" (UID: \"62942548-83af-4767-8de6-1c2b4085ef50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"
Jul 29 16:52:22.240413 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.240267    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62942548-83af-4767-8de6-1c2b4085ef50-tmp\") pod \"klusterlet-addon-workmgr-d9788f-lv2tw\" (UID: \"62942548-83af-4767-8de6-1c2b4085ef50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"
Jul 29 16:52:22.240413 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.240296    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/62942548-83af-4767-8de6-1c2b4085ef50-klusterlet-config\") pod \"klusterlet-addon-workmgr-d9788f-lv2tw\" (UID: \"62942548-83af-4767-8de6-1c2b4085ef50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"
Jul 29 16:52:22.341775 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.341006    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjl2n\" (UniqueName: \"kubernetes.io/projected/62942548-83af-4767-8de6-1c2b4085ef50-kube-api-access-hjl2n\") pod \"klusterlet-addon-workmgr-d9788f-lv2tw\" (UID: \"62942548-83af-4767-8de6-1c2b4085ef50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"
Jul 29 16:52:22.341775 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.341071    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc6l2\" (UniqueName: \"kubernetes.io/projected/162c70b6-13be-43f4-9b50-927f0a934e62-kube-api-access-cc6l2\") pod \"downloads-7c97dc4555-45wpw\" (UID: \"162c70b6-13be-43f4-9b50-927f0a934e62\") " pod="openshift-console/downloads-7c97dc4555-45wpw"
Jul 29 16:52:22.341775 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.341107    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62942548-83af-4767-8de6-1c2b4085ef50-tmp\") pod \"klusterlet-addon-workmgr-d9788f-lv2tw\" (UID: \"62942548-83af-4767-8de6-1c2b4085ef50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"
Jul 29 16:52:22.341775 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.341135    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/62942548-83af-4767-8de6-1c2b4085ef50-klusterlet-config\") pod \"klusterlet-addon-workmgr-d9788f-lv2tw\" (UID: \"62942548-83af-4767-8de6-1c2b4085ef50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"
Jul 29 16:52:22.341775 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.341708    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62942548-83af-4767-8de6-1c2b4085ef50-tmp\") pod \"klusterlet-addon-workmgr-d9788f-lv2tw\" (UID: \"62942548-83af-4767-8de6-1c2b4085ef50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"
Jul 29 16:52:22.346030 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.345995    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/62942548-83af-4767-8de6-1c2b4085ef50-klusterlet-config\") pod \"klusterlet-addon-workmgr-d9788f-lv2tw\" (UID: \"62942548-83af-4767-8de6-1c2b4085ef50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"
Jul 29 16:52:22.357442 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.357403    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjl2n\" (UniqueName: \"kubernetes.io/projected/62942548-83af-4767-8de6-1c2b4085ef50-kube-api-access-hjl2n\") pod \"klusterlet-addon-workmgr-d9788f-lv2tw\" (UID: \"62942548-83af-4767-8de6-1c2b4085ef50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"
Jul 29 16:52:22.372888 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.372840    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pmqs8" event={"ID":"438c3501-4812-489f-ae51-a25e01b9aacc","Type":"ContainerStarted","Data":"30e6a34b87c671746e39f318afcbfebb8d71373fb1ce4a43b36f27188165ab28"}
Jul 29 16:52:22.395600 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.393617    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pmqs8" podStartSLOduration=68.77687152 podStartE2EDuration="1m10.393599695s" totalImagesPullingTime="1.616728175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:22.392667717 +0000 UTC m=+70.882187374" watchObservedRunningTime="2026-07-29 16:52:22.393599695 +0000 UTC m=+70.883119350"
Jul 29 16:52:22.431328 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.431290    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"
Jul 29 16:52:22.442719 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.442650    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc6l2\" (UniqueName: \"kubernetes.io/projected/162c70b6-13be-43f4-9b50-927f0a934e62-kube-api-access-cc6l2\") pod \"downloads-7c97dc4555-45wpw\" (UID: \"162c70b6-13be-43f4-9b50-927f0a934e62\") " pod="openshift-console/downloads-7c97dc4555-45wpw"
Jul 29 16:52:22.462380 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.462341    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc6l2\" (UniqueName: \"kubernetes.io/projected/162c70b6-13be-43f4-9b50-927f0a934e62-kube-api-access-cc6l2\") pod \"downloads-7c97dc4555-45wpw\" (UID: \"162c70b6-13be-43f4-9b50-927f0a934e62\") " pod="openshift-console/downloads-7c97dc4555-45wpw"
Jul 29 16:52:22.505070 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.504928    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-45wpw"
Jul 29 16:52:22.683392 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.682696    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"]
Jul 29 16:52:22.726830 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:22.726795    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-45wpw"]
Jul 29 16:52:23.517256 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:52:23.517220    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62942548_83af_4767_8de6_1c2b4085ef50.slice/crio-a15b681ab6a9d77f6a775abddeb6e2361243f2bc4c246b20e14444b5db675cd6 WatchSource:0}: Error finding container a15b681ab6a9d77f6a775abddeb6e2361243f2bc4c246b20e14444b5db675cd6: Status 404 returned error can't find the container with id a15b681ab6a9d77f6a775abddeb6e2361243f2bc4c246b20e14444b5db675cd6
Jul 29 16:52:24.380305 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:24.380272    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw" event={"ID":"62942548-83af-4767-8de6-1c2b4085ef50","Type":"ContainerStarted","Data":"a15b681ab6a9d77f6a775abddeb6e2361243f2bc4c246b20e14444b5db675cd6"}
Jul 29 16:52:25.074390 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:52:25.074357    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162c70b6_13be_43f4_9b50_927f0a934e62.slice/crio-90340fec92bb7e4137ad7e4ad6d3e1c401aff2f7271ab4edda0e15126d4fa3fb WatchSource:0}: Error finding container 90340fec92bb7e4137ad7e4ad6d3e1c401aff2f7271ab4edda0e15126d4fa3fb: Status 404 returned error can't find the container with id 90340fec92bb7e4137ad7e4ad6d3e1c401aff2f7271ab4edda0e15126d4fa3fb
Jul 29 16:52:25.385495 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:25.385415    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-45wpw" event={"ID":"162c70b6-13be-43f4-9b50-927f0a934e62","Type":"ContainerStarted","Data":"90340fec92bb7e4137ad7e4ad6d3e1c401aff2f7271ab4edda0e15126d4fa3fb"}
Jul 29 16:52:32.427894 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.427814    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dxzh5" event={"ID":"b1bd992f-213e-46bd-87a6-b06f7895719d","Type":"ContainerStarted","Data":"65b4ce12ab3e4231d1b53424f2d17664cfb91bc630ccc91ea44c47f1aa354b0b"}
Jul 29 16:52:32.429881 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.429848    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9" event={"ID":"8d895e18-c64e-4656-868b-4d5191988051","Type":"ContainerStarted","Data":"7c32e545f456592c8d29dd8efe3573f635901470a109bb9558b2e51a695bb4dd"}
Jul 29 16:52:32.432208 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.432030    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-95x46" event={"ID":"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85","Type":"ContainerStarted","Data":"0a931813c3c4b50ddc8f24268a9bcb74e9733ee8ababcf6edf2dff68912ac422"}
Jul 29 16:52:32.432208 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.432086    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-95x46" event={"ID":"b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85","Type":"ContainerStarted","Data":"d48a4e468dcc31c4dd857573b6248353edb575f3f11147019110541fe7461d36"}
Jul 29 16:52:32.432208 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.432143    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-95x46"
Jul 29 16:52:32.434994 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.434899    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7" event={"ID":"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535","Type":"ContainerStarted","Data":"367071b080b1cfb3daed1ece25295b9d84dc2a73b97201a6f60d99864ac14a35"}
Jul 29 16:52:32.434994 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.434958    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7" event={"ID":"ba8efd0d-7b3c-4c24-9443-f7bc8d66b535","Type":"ContainerStarted","Data":"cb508ca0c79348313a158f9bffa2f68840856176a5340333435d56a3fb2e5ecc"}
Jul 29 16:52:32.437580 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.437555    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv" event={"ID":"9a232c9c-55d3-4118-90cc-49aae6db92d5","Type":"ContainerStarted","Data":"17226d4a3af17f8371613e6427a4f3b5ed24c5ac0e461b482d86a247799d8b38"}
Jul 29 16:52:32.440105 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.440061    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fwrgv" event={"ID":"84e4fe40-6165-40b3-9f20-3d3003e57759","Type":"ContainerStarted","Data":"f15bb3686f198cc9b885d8354c60357ebd0a8ce6a62eb344b842274025f6a579"}
Jul 29 16:52:32.441778 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.441713    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw" event={"ID":"62942548-83af-4767-8de6-1c2b4085ef50","Type":"ContainerStarted","Data":"cee8d3233f69b0b78e5241223e7ba45fa3af664a74ec8d282f66e6f61cb54f56"}
Jul 29 16:52:32.441948 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.441925    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"
Jul 29 16:52:32.443952 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.443930    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw"
Jul 29 16:52:32.483354 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.483277    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-787s9" podStartSLOduration=37.621946263 podStartE2EDuration="49.483256991s" totalImagesPullingTime="11.861310728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:32.482282777 +0000 UTC m=+80.971802435" watchObservedRunningTime="2026-07-29 16:52:32.483256991 +0000 UTC m=+80.972776648"
Jul 29 16:52:32.483650 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.483615    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dxzh5" podStartSLOduration=68.184296332 podStartE2EDuration="1m20.483605005s" totalImagesPullingTime="12.299308673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:32.455853481 +0000 UTC m=+80.945373136" watchObservedRunningTime="2026-07-29 16:52:32.483605005 +0000 UTC m=+80.973124661"
Jul 29 16:52:32.514814 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.514755    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-stpc7" podStartSLOduration=37.729653262 podStartE2EDuration="49.514731446s" totalImagesPullingTime="11.785078184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:32.514008936 +0000 UTC m=+81.003528594" watchObservedRunningTime="2026-07-29 16:52:32.514731446 +0000 UTC m=+81.004251104"
Jul 29 16:52:32.535913 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.535853    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9788f-lv2tw" podStartSLOduration=2.266652153 podStartE2EDuration="10.535831059s" totalImagesPullingTime="8.269178906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:32.535276133 +0000 UTC m=+81.024795792" watchObservedRunningTime="2026-07-29 16:52:32.535831059 +0000 UTC m=+81.025350716"
Jul 29 16:52:32.557843 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.557742    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-95x46" podStartSLOduration=36.251409273 podStartE2EDuration="45.557722248s" totalImagesPullingTime="9.306312975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:32.555481418 +0000 UTC m=+81.045001115" watchObservedRunningTime="2026-07-29 16:52:32.557722248 +0000 UTC m=+81.047241904"
Jul 29 16:52:32.574329 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:32.574259    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d59pv" podStartSLOduration=33.567060157 podStartE2EDuration="44.574236379s" totalImagesPullingTime="11.007176222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:32.57285524 +0000 UTC m=+81.062374898" watchObservedRunningTime="2026-07-29 16:52:32.574236379 +0000 UTC m=+81.063756038"
Jul 29 16:52:36.463966 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:36.463875    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fwrgv" event={"ID":"84e4fe40-6165-40b3-9f20-3d3003e57759","Type":"ContainerStarted","Data":"ee8d98fd01a82e75be4256f61f2bff9f7822adcb52100ee1ff85e512cc6dd01c"}
Jul 29 16:52:36.486545 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:36.486490    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fwrgv" podStartSLOduration=16.936690865 podStartE2EDuration="38.486469539s" totalImagesPullingTime="21.549778674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:36.484379692 +0000 UTC m=+84.973899373" watchObservedRunningTime="2026-07-29 16:52:36.486469539 +0000 UTC m=+84.975989198"
Jul 29 16:52:42.380682 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:42.380644    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:52:42.449621 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:42.449588    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-95x46"
Jul 29 16:52:45.014445 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:45.014404    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57b8589c7d-4x49t"]
Jul 29 16:52:45.990442 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:45.990408    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hmkc5"]
Jul 29 16:52:45.994282 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:45.994250    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:45.997620 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:45.997536    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 16:52:45.997835 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:45.997817    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 16:52:45.998079 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:45.998059    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 16:52:45.999086 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:45.999064    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ltjzg\""
Jul 29 16:52:45.999208 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:45.999193    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 16:52:46.048036 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.048000    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-tls\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.048625 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.048599    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78ba15ea-514c-4528-b2e2-392ea1185d0d-sys\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.048845 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.048827    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78ba15ea-514c-4528-b2e2-392ea1185d0d-metrics-client-ca\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.048931 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.048906    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.048995 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.048946    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-wtmp\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.048995 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.048983    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-textfile\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.049096 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.049007    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.049096 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.049043    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/78ba15ea-514c-4528-b2e2-392ea1185d0d-root\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.049096 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.049077    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhjtl\" (UniqueName: \"kubernetes.io/projected/78ba15ea-514c-4528-b2e2-392ea1185d0d-kube-api-access-xhjtl\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.150138 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.150102    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-textfile\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.150138 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.150146    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.150427 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.150201    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/78ba15ea-514c-4528-b2e2-392ea1185d0d-root\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.150427 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.150233    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhjtl\" (UniqueName: \"kubernetes.io/projected/78ba15ea-514c-4528-b2e2-392ea1185d0d-kube-api-access-xhjtl\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.150427 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.150282    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-tls\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.150427 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.150334    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78ba15ea-514c-4528-b2e2-392ea1185d0d-sys\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.150427 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.150390    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78ba15ea-514c-4528-b2e2-392ea1185d0d-metrics-client-ca\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.150653 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.150430    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.150653 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.150464    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-wtmp\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.150653 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:46.150467    2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 16:52:46.150653 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:52:46.150565    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-tls podName:78ba15ea-514c-4528-b2e2-392ea1185d0d nodeName:}" failed. No retries permitted until 2026-07-29 16:52:46.650541203 +0000 UTC m=+95.140060841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-tls") pod "node-exporter-hmkc5" (UID: "78ba15ea-514c-4528-b2e2-392ea1185d0d") : secret "node-exporter-tls" not found
Jul 29 16:52:46.150653 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.150619    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-wtmp\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.150899 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.150810    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78ba15ea-514c-4528-b2e2-392ea1185d0d-sys\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.150899 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.150835    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.151051 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.151027    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-textfile\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.151125 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.151086    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78ba15ea-514c-4528-b2e2-392ea1185d0d-metrics-client-ca\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.151204 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.151137    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/78ba15ea-514c-4528-b2e2-392ea1185d0d-root\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.153215 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.153184    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.160287 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.160260    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhjtl\" (UniqueName: \"kubernetes.io/projected/78ba15ea-514c-4528-b2e2-392ea1185d0d-kube-api-access-xhjtl\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.508265 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.508227    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-45wpw" event={"ID":"162c70b6-13be-43f4-9b50-927f0a934e62","Type":"ContainerStarted","Data":"06f8e47e18170cc1e0ad95608225be765b610eac1f9aa5851b5ba682223c1788"}
Jul 29 16:52:46.508566 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.508542    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-45wpw"
Jul 29 16:52:46.529091 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.529029    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-45wpw" podStartSLOduration=3.857118637 podStartE2EDuration="24.529004314s" totalImagesPullingTime="20.671885677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:46.527029619 +0000 UTC m=+95.016549275" watchObservedRunningTime="2026-07-29 16:52:46.529004314 +0000 UTC m=+95.018523971"
Jul 29 16:52:46.531663 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.531636    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-45wpw"
Jul 29 16:52:46.656231 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.656188    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-tls\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.659180 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.659133    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/78ba15ea-514c-4528-b2e2-392ea1185d0d-node-exporter-tls\") pod \"node-exporter-hmkc5\" (UID: \"78ba15ea-514c-4528-b2e2-392ea1185d0d\") " pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.905314 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:46.905216    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hmkc5"
Jul 29 16:52:46.946222 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:52:46.944913    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ba15ea_514c_4528_b2e2_392ea1185d0d.slice/crio-eaf8cd3bb21d434f52bce7f88c5335e39c771e1e303298f1b61fc5ea770eda4c WatchSource:0}: Error finding container eaf8cd3bb21d434f52bce7f88c5335e39c771e1e303298f1b61fc5ea770eda4c: Status 404 returned error can't find the container with id eaf8cd3bb21d434f52bce7f88c5335e39c771e1e303298f1b61fc5ea770eda4c
Jul 29 16:52:47.517537 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:47.517451    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hmkc5" event={"ID":"78ba15ea-514c-4528-b2e2-392ea1185d0d","Type":"ContainerStarted","Data":"eaf8cd3bb21d434f52bce7f88c5335e39c771e1e303298f1b61fc5ea770eda4c"}
Jul 29 16:52:49.530102 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:49.530061    2570 generic.go:356] "Generic (PLEG): container finished" podID="78ba15ea-514c-4528-b2e2-392ea1185d0d" containerID="1eaef09f74018e9d7f73ba7f1a0dff3e6cb3754476301ba5fa5aaf40c4606373" exitCode=0
Jul 29 16:52:49.530684 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:49.530158    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hmkc5" event={"ID":"78ba15ea-514c-4528-b2e2-392ea1185d0d","Type":"ContainerDied","Data":"1eaef09f74018e9d7f73ba7f1a0dff3e6cb3754476301ba5fa5aaf40c4606373"}
Jul 29 16:52:50.538123 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:50.538077    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hmkc5" event={"ID":"78ba15ea-514c-4528-b2e2-392ea1185d0d","Type":"ContainerStarted","Data":"1a2159672a63388e17ca6ee0eb687e83513ee78b29155663ef28f2f385aa6485"}
Jul 29 16:52:50.538718 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:50.538151    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hmkc5" event={"ID":"78ba15ea-514c-4528-b2e2-392ea1185d0d","Type":"ContainerStarted","Data":"a6a317fa1565598d27d89c856c8c60f6d3c2246e75cad31d7949ff8625aa4bfc"}
Jul 29 16:52:50.563409 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:50.563348    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hmkc5" podStartSLOduration=3.878878625 podStartE2EDuration="5.563315433s" totalImagesPullingTime="1.588697008s" totalInitContainerRuntime="95.7398ms" isStatefulPod=true podCreationTimestamp="2026-07-29 16:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:50.561965614 +0000 UTC m=+99.051485271" watchObservedRunningTime="2026-07-29 16:52:50.563315433 +0000 UTC m=+99.052835102"
Jul 29 16:52:51.365814 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:51.365779    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zb222"
Jul 29 16:52:55.693058 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.693021    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d56d678dc-pkprr"]
Jul 29 16:52:55.727466 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.727432    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d56d678dc-pkprr"]
Jul 29 16:52:55.727629 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.727550    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.730381 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.730347    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 16:52:55.730535 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.730490    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 16:52:55.731604 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.731580    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 16:52:55.731776 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.731613    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 16:52:55.731943 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.731680    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 16:52:55.732947 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.732928    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ktffr\""
Jul 29 16:52:55.737036 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.737017    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 16:52:55.838519 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.838480    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-config\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.838519 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.838528    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-serving-cert\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.838762 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.838555    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-oauth-serving-cert\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.838762 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.838611    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-oauth-config\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.838762 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.838630    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-trusted-ca-bundle\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.838762 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.838663    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-service-ca\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.838762 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.838689    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zwng\" (UniqueName: \"kubernetes.io/projected/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-kube-api-access-9zwng\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.940081 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.940044    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-config\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.940081 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.940095    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-serving-cert\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.940423 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.940127    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-oauth-serving-cert\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.940423 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.940149    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-oauth-config\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.940423 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.940185    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-trusted-ca-bundle\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.940423 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.940215    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-service-ca\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.940423 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.940237    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zwng\" (UniqueName: \"kubernetes.io/projected/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-kube-api-access-9zwng\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.940952 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.940900    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-service-ca\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.940952 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.940925    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-config\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.941085 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.940958    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-oauth-serving-cert\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.941478 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.941455    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-trusted-ca-bundle\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.942692 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.942672    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-serving-cert\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.942752 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.942727    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-oauth-config\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:55.949804 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:55.949737    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zwng\" (UniqueName: \"kubernetes.io/projected/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-kube-api-access-9zwng\") pod \"console-7d56d678dc-pkprr\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") " pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:56.038691 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:56.038646    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:52:56.203696 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:56.203668    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d56d678dc-pkprr"]
Jul 29 16:52:56.206177 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:52:56.206142    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0f49c5_e9a8_4e75_b126_0c3cfb0ccea5.slice/crio-31d38805bc77fde4298de96d816a9dbef13655dabace66395fe5afe966d4f821 WatchSource:0}: Error finding container 31d38805bc77fde4298de96d816a9dbef13655dabace66395fe5afe966d4f821: Status 404 returned error can't find the container with id 31d38805bc77fde4298de96d816a9dbef13655dabace66395fe5afe966d4f821
Jul 29 16:52:56.564858 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:52:56.564824    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d56d678dc-pkprr" event={"ID":"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5","Type":"ContainerStarted","Data":"31d38805bc77fde4298de96d816a9dbef13655dabace66395fe5afe966d4f821"}
Jul 29 16:53:00.582777 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:00.582734    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d56d678dc-pkprr" event={"ID":"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5","Type":"ContainerStarted","Data":"dc9fc894e9e17c2e1109701b3dd3d883dce95ece5947d06013e2e990b42b8291"}
Jul 29 16:53:00.600576 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:00.600524    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d56d678dc-pkprr" podStartSLOduration=2.122780873 podStartE2EDuration="5.600507914s" totalImagesPullingTime="3.477727041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:53:00.6002495 +0000 UTC m=+109.089769157" watchObservedRunningTime="2026-07-29 16:53:00.600507914 +0000 UTC m=+109.090027569"
Jul 29 16:53:02.592396 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:02.592361    2570 generic.go:356] "Generic (PLEG): container finished" podID="af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e" containerID="02481bcb5d6bc4038c352af4cc3bc816166b8b55092664f028649dcaf03470bb" exitCode=0
Jul 29 16:53:02.592776 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:02.592425    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zx4jr" event={"ID":"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e","Type":"ContainerDied","Data":"02481bcb5d6bc4038c352af4cc3bc816166b8b55092664f028649dcaf03470bb"}
Jul 29 16:53:02.592776 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:02.592732    2570 scope.go:117] "RemoveContainer" containerID="02481bcb5d6bc4038c352af4cc3bc816166b8b55092664f028649dcaf03470bb"
Jul 29 16:53:03.598071 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:03.598040    2570 generic.go:356] "Generic (PLEG): container finished" podID="2c5076ed-8a8a-4866-801a-5b0037ec4d36" containerID="7334e741781ce8b9b0c6f8090b475692503fe9a8ddfa733484309d897c2e6d22" exitCode=0
Jul 29 16:53:03.598533 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:03.598115    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm" event={"ID":"2c5076ed-8a8a-4866-801a-5b0037ec4d36","Type":"ContainerDied","Data":"7334e741781ce8b9b0c6f8090b475692503fe9a8ddfa733484309d897c2e6d22"}
Jul 29 16:53:03.598533 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:03.598485    2570 scope.go:117] "RemoveContainer" containerID="7334e741781ce8b9b0c6f8090b475692503fe9a8ddfa733484309d897c2e6d22"
Jul 29 16:53:03.600129 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:03.600108    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zx4jr" event={"ID":"af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e","Type":"ContainerStarted","Data":"410e74369003be9cb7b58627869f842ac2b78dbc609fd98fd9cb1d55974caac8"}
Jul 29 16:53:04.605485 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:04.605452    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7httm" event={"ID":"2c5076ed-8a8a-4866-801a-5b0037ec4d36","Type":"ContainerStarted","Data":"225a7b9f716e01c6fc5b5efb40d1bfc8dbecd6b4da3b419da24ad11c3c18d760"}
Jul 29 16:53:06.039248 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:06.039205    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:53:06.039757 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:06.039273    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:53:06.044021 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:06.043996    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:53:06.619663 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:06.619634    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:53:07.620954 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:07.620921    2570 generic.go:356] "Generic (PLEG): container finished" podID="a86ddea8-14e1-4ea5-bd16-20e267ef74f1" containerID="2674d464dccff6e54fa97cab2e2b3b2e0011e912259cb31bbf3f13b0aec7c874" exitCode=0
Jul 29 16:53:07.621373 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:07.621000    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn" event={"ID":"a86ddea8-14e1-4ea5-bd16-20e267ef74f1","Type":"ContainerDied","Data":"2674d464dccff6e54fa97cab2e2b3b2e0011e912259cb31bbf3f13b0aec7c874"}
Jul 29 16:53:07.621431 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:07.621410    2570 scope.go:117] "RemoveContainer" containerID="2674d464dccff6e54fa97cab2e2b3b2e0011e912259cb31bbf3f13b0aec7c874"
Jul 29 16:53:08.626921 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:08.626886    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fb7nn" event={"ID":"a86ddea8-14e1-4ea5-bd16-20e267ef74f1","Type":"ContainerStarted","Data":"40d26e8162f9ad3a2b343a2971b11220925456eab24dfd3a086ff711aea96e21"}
Jul 29 16:53:10.034011 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.033968    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57b8589c7d-4x49t" podUID="1a5e4084-948e-4a3b-9752-b80d169c5078" containerName="registry" containerID="cri-o://6eafa9f25e1abba92d5a6cc01a55cf64d1bd99b2921625e0d7cd8fa2a873c280" gracePeriod=30
Jul 29 16:53:10.329060 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.329030    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:53:10.463326 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.463294    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1a5e4084-948e-4a3b-9752-b80d169c5078-ca-trust-extracted\") pod \"1a5e4084-948e-4a3b-9752-b80d169c5078\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") "
Jul 29 16:53:10.463326 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.463334    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1a5e4084-948e-4a3b-9752-b80d169c5078-installation-pull-secrets\") pod \"1a5e4084-948e-4a3b-9752-b80d169c5078\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") "
Jul 29 16:53:10.463590 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.463368    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls\") pod \"1a5e4084-948e-4a3b-9752-b80d169c5078\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") "
Jul 29 16:53:10.463590 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.463391    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-certificates\") pod \"1a5e4084-948e-4a3b-9752-b80d169c5078\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") "
Jul 29 16:53:10.463590 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.463414    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1a5e4084-948e-4a3b-9752-b80d169c5078-image-registry-private-configuration\") pod \"1a5e4084-948e-4a3b-9752-b80d169c5078\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") "
Jul 29 16:53:10.463590 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.463469    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a5e4084-948e-4a3b-9752-b80d169c5078-trusted-ca\") pod \"1a5e4084-948e-4a3b-9752-b80d169c5078\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") "
Jul 29 16:53:10.463590 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.463505    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-bound-sa-token\") pod \"1a5e4084-948e-4a3b-9752-b80d169c5078\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") "
Jul 29 16:53:10.463590 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.463538    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvx47\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-kube-api-access-hvx47\") pod \"1a5e4084-948e-4a3b-9752-b80d169c5078\" (UID: \"1a5e4084-948e-4a3b-9752-b80d169c5078\") "
Jul 29 16:53:10.463898 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.463835    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1a5e4084-948e-4a3b-9752-b80d169c5078" (UID: "1a5e4084-948e-4a3b-9752-b80d169c5078"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 16:53:10.464302 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.464238    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5e4084-948e-4a3b-9752-b80d169c5078-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1a5e4084-948e-4a3b-9752-b80d169c5078" (UID: "1a5e4084-948e-4a3b-9752-b80d169c5078"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 16:53:10.466266 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.466228    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5e4084-948e-4a3b-9752-b80d169c5078-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1a5e4084-948e-4a3b-9752-b80d169c5078" (UID: "1a5e4084-948e-4a3b-9752-b80d169c5078"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 16:53:10.466424 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.466387    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1a5e4084-948e-4a3b-9752-b80d169c5078" (UID: "1a5e4084-948e-4a3b-9752-b80d169c5078"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:53:10.466524 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.466398    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-kube-api-access-hvx47" (OuterVolumeSpecName: "kube-api-access-hvx47") pod "1a5e4084-948e-4a3b-9752-b80d169c5078" (UID: "1a5e4084-948e-4a3b-9752-b80d169c5078"). InnerVolumeSpecName "kube-api-access-hvx47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:53:10.466524 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.466459    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5e4084-948e-4a3b-9752-b80d169c5078-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1a5e4084-948e-4a3b-9752-b80d169c5078" (UID: "1a5e4084-948e-4a3b-9752-b80d169c5078"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 16:53:10.466524 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.466491    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1a5e4084-948e-4a3b-9752-b80d169c5078" (UID: "1a5e4084-948e-4a3b-9752-b80d169c5078"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:53:10.472069 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.472039    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5e4084-948e-4a3b-9752-b80d169c5078-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1a5e4084-948e-4a3b-9752-b80d169c5078" (UID: "1a5e4084-948e-4a3b-9752-b80d169c5078"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:53:10.565220 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.565105    2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:53:10.565220 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.565134    2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1a5e4084-948e-4a3b-9752-b80d169c5078-registry-certificates\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:53:10.565220 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.565145    2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1a5e4084-948e-4a3b-9752-b80d169c5078-image-registry-private-configuration\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:53:10.565220 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.565155    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a5e4084-948e-4a3b-9752-b80d169c5078-trusted-ca\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:53:10.565220 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.565176    2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-bound-sa-token\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:53:10.565220 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.565184    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvx47\" (UniqueName: \"kubernetes.io/projected/1a5e4084-948e-4a3b-9752-b80d169c5078-kube-api-access-hvx47\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:53:10.565220 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.565192    2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1a5e4084-948e-4a3b-9752-b80d169c5078-ca-trust-extracted\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:53:10.565220 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.565200    2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1a5e4084-948e-4a3b-9752-b80d169c5078-installation-pull-secrets\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:53:10.636490 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.636458    2570 generic.go:356] "Generic (PLEG): container finished" podID="1a5e4084-948e-4a3b-9752-b80d169c5078" containerID="6eafa9f25e1abba92d5a6cc01a55cf64d1bd99b2921625e0d7cd8fa2a873c280" exitCode=0
Jul 29 16:53:10.636657 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.636498    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b8589c7d-4x49t"
Jul 29 16:53:10.636657 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.636502    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b8589c7d-4x49t" event={"ID":"1a5e4084-948e-4a3b-9752-b80d169c5078","Type":"ContainerDied","Data":"6eafa9f25e1abba92d5a6cc01a55cf64d1bd99b2921625e0d7cd8fa2a873c280"}
Jul 29 16:53:10.636657 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.636604    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b8589c7d-4x49t" event={"ID":"1a5e4084-948e-4a3b-9752-b80d169c5078","Type":"ContainerDied","Data":"0efc1fabc7663345ebaa2bb4c95d5501fda1b811671e1fcdb176d1a189561e9d"}
Jul 29 16:53:10.636657 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.636625    2570 scope.go:117] "RemoveContainer" containerID="6eafa9f25e1abba92d5a6cc01a55cf64d1bd99b2921625e0d7cd8fa2a873c280"
Jul 29 16:53:10.645241 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.645222    2570 scope.go:117] "RemoveContainer" containerID="6eafa9f25e1abba92d5a6cc01a55cf64d1bd99b2921625e0d7cd8fa2a873c280"
Jul 29 16:53:10.645548 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:53:10.645530    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eafa9f25e1abba92d5a6cc01a55cf64d1bd99b2921625e0d7cd8fa2a873c280\": container with ID starting with 6eafa9f25e1abba92d5a6cc01a55cf64d1bd99b2921625e0d7cd8fa2a873c280 not found: ID does not exist" containerID="6eafa9f25e1abba92d5a6cc01a55cf64d1bd99b2921625e0d7cd8fa2a873c280"
Jul 29 16:53:10.645624 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.645557    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eafa9f25e1abba92d5a6cc01a55cf64d1bd99b2921625e0d7cd8fa2a873c280"} err="failed to get container status \"6eafa9f25e1abba92d5a6cc01a55cf64d1bd99b2921625e0d7cd8fa2a873c280\": rpc error: code = NotFound desc = could not find container \"6eafa9f25e1abba92d5a6cc01a55cf64d1bd99b2921625e0d7cd8fa2a873c280\": container with ID starting with 6eafa9f25e1abba92d5a6cc01a55cf64d1bd99b2921625e0d7cd8fa2a873c280 not found: ID does not exist"
Jul 29 16:53:10.659666 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.658956    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57b8589c7d-4x49t"]
Jul 29 16:53:10.663972 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:10.663944    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57b8589c7d-4x49t"]
Jul 29 16:53:11.912739 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:53:11.912701    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a5e4084-948e-4a3b-9752-b80d169c5078" path="/var/lib/kubelet/pods/1a5e4084-948e-4a3b-9752-b80d169c5078/volumes"
Jul 29 16:54:08.602455 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.602425    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5859d65dc9-4hlnh"]
Jul 29 16:54:08.602911 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.602729    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a5e4084-948e-4a3b-9752-b80d169c5078" containerName="registry"
Jul 29 16:54:08.602911 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.602742    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a5e4084-948e-4a3b-9752-b80d169c5078" containerName="registry"
Jul 29 16:54:08.602911 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.602807    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a5e4084-948e-4a3b-9752-b80d169c5078" containerName="registry"
Jul 29 16:54:08.605899 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.605876    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.634267 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.634231    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5859d65dc9-4hlnh"]
Jul 29 16:54:08.712753 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.712717    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlktj\" (UniqueName: \"kubernetes.io/projected/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-kube-api-access-mlktj\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.712913 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.712758    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-serving-cert\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.712913 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.712781    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-service-ca\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.712913 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.712854    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-config\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.712913 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.712902    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-trusted-ca-bundle\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.713055 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.712922    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-oauth-serving-cert\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.713055 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.713007    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-oauth-config\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.813623 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.813584    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-serving-cert\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.813623 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.813623    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-service-ca\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.813856 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.813650    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-config\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.813856 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.813687    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-trusted-ca-bundle\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.813856 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.813711    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-oauth-serving-cert\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.813856 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.813739    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-oauth-config\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.813856 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.813819    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlktj\" (UniqueName: \"kubernetes.io/projected/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-kube-api-access-mlktj\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.814535 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.814510    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-config\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.814667 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.814534    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-service-ca\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.814667 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.814636    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-oauth-serving-cert\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.814783 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.814639    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-trusted-ca-bundle\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.816222 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.816203    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-serving-cert\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.816388 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.816366    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-oauth-config\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.825378 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.825357    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlktj\" (UniqueName: \"kubernetes.io/projected/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-kube-api-access-mlktj\") pod \"console-5859d65dc9-4hlnh\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") " pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:08.915621 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:08.915536    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:09.068456 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:09.068424    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5859d65dc9-4hlnh"]
Jul 29 16:54:09.071542 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:54:09.071518    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6a2eeb_e489_4940_a5c2_f845e807b4b1.slice/crio-c3cf104614226d66fa31a9a75ee0ee769d268ae6f49305dd38f232ee02f52c5b WatchSource:0}: Error finding container c3cf104614226d66fa31a9a75ee0ee769d268ae6f49305dd38f232ee02f52c5b: Status 404 returned error can't find the container with id c3cf104614226d66fa31a9a75ee0ee769d268ae6f49305dd38f232ee02f52c5b
Jul 29 16:54:09.857496 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:09.857458    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5859d65dc9-4hlnh" event={"ID":"aa6a2eeb-e489-4940-a5c2-f845e807b4b1","Type":"ContainerStarted","Data":"57c500ed14184aad426cf5ea29298e1ac0f21810e7bf78f3d54fbd12bced1585"}
Jul 29 16:54:09.857889 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:09.857550    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5859d65dc9-4hlnh" event={"ID":"aa6a2eeb-e489-4940-a5c2-f845e807b4b1","Type":"ContainerStarted","Data":"c3cf104614226d66fa31a9a75ee0ee769d268ae6f49305dd38f232ee02f52c5b"}
Jul 29 16:54:09.895832 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:09.895778    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5859d65dc9-4hlnh" podStartSLOduration=1.89576017 podStartE2EDuration="1.89576017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:54:09.893655275 +0000 UTC m=+178.383174931" watchObservedRunningTime="2026-07-29 16:54:09.89576017 +0000 UTC m=+178.385279884"
Jul 29 16:54:18.915938 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:18.915886    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:18.916503 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:18.916246    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:18.920904 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:18.920881    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:19.898077 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:19.898045    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:54:19.962585 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:19.962545    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d56d678dc-pkprr"]
Jul 29 16:54:44.983644 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:44.983582    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d56d678dc-pkprr" podUID="5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5" containerName="console" containerID="cri-o://dc9fc894e9e17c2e1109701b3dd3d883dce95ece5947d06013e2e990b42b8291" gracePeriod=15
Jul 29 16:54:45.230313 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.230288    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:54:45.312818 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.312781    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-service-ca\") pod \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") "
Jul 29 16:54:45.312818 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.312833    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-serving-cert\") pod \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") "
Jul 29 16:54:45.313093 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.312851    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zwng\" (UniqueName: \"kubernetes.io/projected/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-kube-api-access-9zwng\") pod \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") "
Jul 29 16:54:45.313093 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.312877    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-trusted-ca-bundle\") pod \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") "
Jul 29 16:54:45.313093 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.313002    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-oauth-serving-cert\") pod \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") "
Jul 29 16:54:45.313093 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.313039    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-oauth-config\") pod \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") "
Jul 29 16:54:45.313093 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.313065    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-config\") pod \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\" (UID: \"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5\") "
Jul 29 16:54:45.313375 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.313336    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-service-ca" (OuterVolumeSpecName: "service-ca") pod "5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5" (UID: "5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 16:54:45.313375 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.313342    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5" (UID: "5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 16:54:45.313569 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.313442    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5" (UID: "5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 16:54:45.313640 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.313606    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-config" (OuterVolumeSpecName: "console-config") pod "5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5" (UID: "5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 16:54:45.315303 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.315268    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-kube-api-access-9zwng" (OuterVolumeSpecName: "kube-api-access-9zwng") pod "5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5" (UID: "5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5"). InnerVolumeSpecName "kube-api-access-9zwng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:54:45.315303 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.315280    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5" (UID: "5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 16:54:45.315448 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.315372    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5" (UID: "5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 16:54:45.413673 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.413635    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-service-ca\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:54:45.413673 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.413663    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-serving-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:54:45.413673 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.413674    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zwng\" (UniqueName: \"kubernetes.io/projected/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-kube-api-access-9zwng\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:54:45.413673 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.413682    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-trusted-ca-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:54:45.413919 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.413691    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-oauth-serving-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:54:45.413919 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.413700    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-oauth-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:54:45.413919 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.413708    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5-console-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:54:45.990036 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.990002    2570 generic.go:356] "Generic (PLEG): container finished" podID="5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5" containerID="dc9fc894e9e17c2e1109701b3dd3d883dce95ece5947d06013e2e990b42b8291" exitCode=0
Jul 29 16:54:45.990455 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.990046    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d56d678dc-pkprr" event={"ID":"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5","Type":"ContainerDied","Data":"dc9fc894e9e17c2e1109701b3dd3d883dce95ece5947d06013e2e990b42b8291"}
Jul 29 16:54:45.990455 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.990070    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d56d678dc-pkprr"
Jul 29 16:54:45.990455 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.990097    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d56d678dc-pkprr" event={"ID":"5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5","Type":"ContainerDied","Data":"31d38805bc77fde4298de96d816a9dbef13655dabace66395fe5afe966d4f821"}
Jul 29 16:54:45.990455 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.990142    2570 scope.go:117] "RemoveContainer" containerID="dc9fc894e9e17c2e1109701b3dd3d883dce95ece5947d06013e2e990b42b8291"
Jul 29 16:54:45.998079 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.998058    2570 scope.go:117] "RemoveContainer" containerID="dc9fc894e9e17c2e1109701b3dd3d883dce95ece5947d06013e2e990b42b8291"
Jul 29 16:54:45.998384 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:54:45.998364    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9fc894e9e17c2e1109701b3dd3d883dce95ece5947d06013e2e990b42b8291\": container with ID starting with dc9fc894e9e17c2e1109701b3dd3d883dce95ece5947d06013e2e990b42b8291 not found: ID does not exist" containerID="dc9fc894e9e17c2e1109701b3dd3d883dce95ece5947d06013e2e990b42b8291"
Jul 29 16:54:45.998452 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:45.998395    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9fc894e9e17c2e1109701b3dd3d883dce95ece5947d06013e2e990b42b8291"} err="failed to get container status \"dc9fc894e9e17c2e1109701b3dd3d883dce95ece5947d06013e2e990b42b8291\": rpc error: code = NotFound desc = could not find container \"dc9fc894e9e17c2e1109701b3dd3d883dce95ece5947d06013e2e990b42b8291\": container with ID starting with dc9fc894e9e17c2e1109701b3dd3d883dce95ece5947d06013e2e990b42b8291 not found: ID does not exist"
Jul 29 16:54:46.008588 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:46.008562    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d56d678dc-pkprr"]
Jul 29 16:54:46.012073 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:46.012053    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d56d678dc-pkprr"]
Jul 29 16:54:47.912561 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:54:47.912533    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5" path="/var/lib/kubelet/pods/5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5/volumes"
Jul 29 16:55:50.784804 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.784766    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"]
Jul 29 16:55:50.785242 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.785065    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5" containerName="console"
Jul 29 16:55:50.785242 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.785076    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5" containerName="console"
Jul 29 16:55:50.785242 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.785122    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b0f49c5-e9a8-4e75-b126-0c3cfb0ccea5" containerName="console"
Jul 29 16:55:50.788257 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.788239    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"
Jul 29 16:55:50.790644 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.790615    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 16:55:50.790779 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.790676    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 16:55:50.791473 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.791454    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-tkjsw\""
Jul 29 16:55:50.796746 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.796723    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"]
Jul 29 16:55:50.827916 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.827873    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq\" (UID: \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"
Jul 29 16:55:50.828132 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.827982    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq\" (UID: \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"
Jul 29 16:55:50.828132 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.828047    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4tww\" (UniqueName: \"kubernetes.io/projected/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-kube-api-access-p4tww\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq\" (UID: \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"
Jul 29 16:55:50.929117 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.929014    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq\" (UID: \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"
Jul 29 16:55:50.929117 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.929096    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tww\" (UniqueName: \"kubernetes.io/projected/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-kube-api-access-p4tww\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq\" (UID: \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"
Jul 29 16:55:50.929117 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.929128    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq\" (UID: \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"
Jul 29 16:55:50.929523 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.929500    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq\" (UID: \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"
Jul 29 16:55:50.929589 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.929546    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq\" (UID: \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"
Jul 29 16:55:50.938211 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:50.938187    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4tww\" (UniqueName: \"kubernetes.io/projected/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-kube-api-access-p4tww\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq\" (UID: \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"
Jul 29 16:55:51.098656 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:51.098614    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"
Jul 29 16:55:51.235779 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:51.235753    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"]
Jul 29 16:55:51.238127 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:55:51.238101    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6182ec00_4ac2_41a5_9bb0_7d545fd0ccd0.slice/crio-f5e09b701636a18864e096bf3850da3c250413d798c456e8d789e65f473fb659 WatchSource:0}: Error finding container f5e09b701636a18864e096bf3850da3c250413d798c456e8d789e65f473fb659: Status 404 returned error can't find the container with id f5e09b701636a18864e096bf3850da3c250413d798c456e8d789e65f473fb659
Jul 29 16:55:52.235633 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:52.235592    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq" event={"ID":"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0","Type":"ContainerStarted","Data":"f5e09b701636a18864e096bf3850da3c250413d798c456e8d789e65f473fb659"}
Jul 29 16:55:58.261405 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:58.261371    2570 generic.go:356] "Generic (PLEG): container finished" podID="6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0" containerID="0bc89cb1967cbb43c58401bfe5493656dc01d92017071035ce276de1bd5132f6" exitCode=0
Jul 29 16:55:58.261811 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:55:58.261447    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq" event={"ID":"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0","Type":"ContainerDied","Data":"0bc89cb1967cbb43c58401bfe5493656dc01d92017071035ce276de1bd5132f6"}
Jul 29 16:56:01.274724 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:01.274690    2570 generic.go:356] "Generic (PLEG): container finished" podID="6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0" containerID="a08fc19f85586432dc72a49c0a28fe767a89f572402557633988967a604b81f3" exitCode=0
Jul 29 16:56:01.275101 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:01.274748    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq" event={"ID":"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0","Type":"ContainerDied","Data":"a08fc19f85586432dc72a49c0a28fe767a89f572402557633988967a604b81f3"}
Jul 29 16:56:10.313375 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:10.313343    2570 generic.go:356] "Generic (PLEG): container finished" podID="6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0" containerID="c5ae0fbc4e24db703f9caacf1180dd5661e1e020b3e24fd810f5ed3c09fb6710" exitCode=0
Jul 29 16:56:10.313779 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:10.313394    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq" event={"ID":"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0","Type":"ContainerDied","Data":"c5ae0fbc4e24db703f9caacf1180dd5661e1e020b3e24fd810f5ed3c09fb6710"}
Jul 29 16:56:11.442036 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:11.442008    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"
Jul 29 16:56:11.504225 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:11.504192    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-util\") pod \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\" (UID: \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\") "
Jul 29 16:56:11.504382 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:11.504283    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-bundle\") pod \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\" (UID: \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\") "
Jul 29 16:56:11.504382 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:11.504351    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4tww\" (UniqueName: \"kubernetes.io/projected/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-kube-api-access-p4tww\") pod \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\" (UID: \"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0\") "
Jul 29 16:56:11.504942 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:11.504914    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-bundle" (OuterVolumeSpecName: "bundle") pod "6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0" (UID: "6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:56:11.506636 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:11.506611    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-kube-api-access-p4tww" (OuterVolumeSpecName: "kube-api-access-p4tww") pod "6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0" (UID: "6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0"). InnerVolumeSpecName "kube-api-access-p4tww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:56:11.508418 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:11.508397    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-util" (OuterVolumeSpecName: "util") pod "6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0" (UID: "6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:56:11.605225 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:11.605110    2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:56:11.605225 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:11.605152    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4tww\" (UniqueName: \"kubernetes.io/projected/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-kube-api-access-p4tww\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:56:11.605225 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:11.605183    2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:56:11.810107 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:11.810080    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 16:56:11.811357 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:11.811334    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 16:56:11.819899 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:11.819879    2570 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 16:56:12.323547 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:12.323513    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq" event={"ID":"6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0","Type":"ContainerDied","Data":"f5e09b701636a18864e096bf3850da3c250413d798c456e8d789e65f473fb659"}
Jul 29 16:56:12.323547 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:12.323534    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5m9tdq"
Jul 29 16:56:12.323772 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:12.323585    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5e09b701636a18864e096bf3850da3c250413d798c456e8d789e65f473fb659"
Jul 29 16:56:19.912881 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.912845    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59"]
Jul 29 16:56:19.913291 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.913092    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0" containerName="extract"
Jul 29 16:56:19.913291 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.913103    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0" containerName="extract"
Jul 29 16:56:19.913291 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.913111    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0" containerName="pull"
Jul 29 16:56:19.913291 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.913116    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0" containerName="pull"
Jul 29 16:56:19.913291 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.913128    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0" containerName="util"
Jul 29 16:56:19.913291 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.913133    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0" containerName="util"
Jul 29 16:56:19.913291 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.913200    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6182ec00-4ac2-41a5-9bb0-7d545fd0ccd0" containerName="extract"
Jul 29 16:56:19.918291 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.918262    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59"
Jul 29 16:56:19.920889 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.920859    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-zfrxs\""
Jul 29 16:56:19.921006 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.920934    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 29 16:56:19.921607 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.921591    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 29 16:56:19.921743 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.921728    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 29 16:56:19.929542 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.929520    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59"]
Jul 29 16:56:19.966965 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.966929    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f0c7e152-b493-41c9-be9b-ac9c7162b857-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-drr59\" (UID: \"f0c7e152-b493-41c9-be9b-ac9c7162b857\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59"
Jul 29 16:56:19.967138 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:19.966998    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ths7t\" (UniqueName: \"kubernetes.io/projected/f0c7e152-b493-41c9-be9b-ac9c7162b857-kube-api-access-ths7t\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-drr59\" (UID: \"f0c7e152-b493-41c9-be9b-ac9c7162b857\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59"
Jul 29 16:56:20.067452 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:20.067418    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f0c7e152-b493-41c9-be9b-ac9c7162b857-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-drr59\" (UID: \"f0c7e152-b493-41c9-be9b-ac9c7162b857\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59"
Jul 29 16:56:20.067625 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:20.067488    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ths7t\" (UniqueName: \"kubernetes.io/projected/f0c7e152-b493-41c9-be9b-ac9c7162b857-kube-api-access-ths7t\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-drr59\" (UID: \"f0c7e152-b493-41c9-be9b-ac9c7162b857\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59"
Jul 29 16:56:20.069734 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:20.069712    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f0c7e152-b493-41c9-be9b-ac9c7162b857-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-drr59\" (UID: \"f0c7e152-b493-41c9-be9b-ac9c7162b857\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59"
Jul 29 16:56:20.079890 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:20.079854    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ths7t\" (UniqueName: \"kubernetes.io/projected/f0c7e152-b493-41c9-be9b-ac9c7162b857-kube-api-access-ths7t\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-drr59\" (UID: \"f0c7e152-b493-41c9-be9b-ac9c7162b857\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59"
Jul 29 16:56:20.228869 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:20.228783    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59"
Jul 29 16:56:20.361680 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:20.361657    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59"]
Jul 29 16:56:20.364366 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:56:20.364337    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c7e152_b493_41c9_be9b_ac9c7162b857.slice/crio-b79e7d961a6920df2d18c1077e72cf824c9157181bcb7cd7e24bc47a53185251 WatchSource:0}: Error finding container b79e7d961a6920df2d18c1077e72cf824c9157181bcb7cd7e24bc47a53185251: Status 404 returned error can't find the container with id b79e7d961a6920df2d18c1077e72cf824c9157181bcb7cd7e24bc47a53185251
Jul 29 16:56:20.366635 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:20.366618    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 16:56:21.359205 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:21.359138    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59" event={"ID":"f0c7e152-b493-41c9-be9b-ac9c7162b857","Type":"ContainerStarted","Data":"b79e7d961a6920df2d18c1077e72cf824c9157181bcb7cd7e24bc47a53185251"}
Jul 29 16:56:23.369366 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:23.369331    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59" event={"ID":"f0c7e152-b493-41c9-be9b-ac9c7162b857","Type":"ContainerStarted","Data":"9aa46e2160edc03063397290e8959a816217420fdcc0f8fa9842a6e2a3ac52c5"}
Jul 29 16:56:23.369755 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:23.369429    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59"
Jul 29 16:56:23.390687 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:23.390637    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59" podStartSLOduration=1.495411208 podStartE2EDuration="4.390620916s" totalImagesPullingTime="2.895209708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:56:23.388922523 +0000 UTC m=+311.878442179" watchObservedRunningTime="2026-07-29 16:56:23.390620916 +0000 UTC m=+311.880140573"
Jul 29 16:56:24.015738 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.015702    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-jrgg9"]
Jul 29 16:56:24.019364 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.019346    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-jrgg9"
Jul 29 16:56:24.021977 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.021943    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 29 16:56:24.022118 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.021952    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-bn7k2\""
Jul 29 16:56:24.022118 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.021981    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 29 16:56:24.028695 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.028668    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-jrgg9"]
Jul 29 16:56:24.103472 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.103434    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/633c8dd1-fc22-44c7-9b77-3ec14af9d4c0-cabundle0\") pod \"keda-operator-77f648d694-jrgg9\" (UID: \"633c8dd1-fc22-44c7-9b77-3ec14af9d4c0\") " pod="openshift-keda/keda-operator-77f648d694-jrgg9"
Jul 29 16:56:24.103659 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.103563    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/633c8dd1-fc22-44c7-9b77-3ec14af9d4c0-certificates\") pod \"keda-operator-77f648d694-jrgg9\" (UID: \"633c8dd1-fc22-44c7-9b77-3ec14af9d4c0\") " pod="openshift-keda/keda-operator-77f648d694-jrgg9"
Jul 29 16:56:24.103659 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.103594    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfndk\" (UniqueName: \"kubernetes.io/projected/633c8dd1-fc22-44c7-9b77-3ec14af9d4c0-kube-api-access-bfndk\") pod \"keda-operator-77f648d694-jrgg9\" (UID: \"633c8dd1-fc22-44c7-9b77-3ec14af9d4c0\") " pod="openshift-keda/keda-operator-77f648d694-jrgg9"
Jul 29 16:56:24.204967 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.204932    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/633c8dd1-fc22-44c7-9b77-3ec14af9d4c0-cabundle0\") pod \"keda-operator-77f648d694-jrgg9\" (UID: \"633c8dd1-fc22-44c7-9b77-3ec14af9d4c0\") " pod="openshift-keda/keda-operator-77f648d694-jrgg9"
Jul 29 16:56:24.205116 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.204997    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/633c8dd1-fc22-44c7-9b77-3ec14af9d4c0-certificates\") pod \"keda-operator-77f648d694-jrgg9\" (UID: \"633c8dd1-fc22-44c7-9b77-3ec14af9d4c0\") " pod="openshift-keda/keda-operator-77f648d694-jrgg9"
Jul 29 16:56:24.205116 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.205021    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfndk\" (UniqueName: \"kubernetes.io/projected/633c8dd1-fc22-44c7-9b77-3ec14af9d4c0-kube-api-access-bfndk\") pod \"keda-operator-77f648d694-jrgg9\" (UID: \"633c8dd1-fc22-44c7-9b77-3ec14af9d4c0\") " pod="openshift-keda/keda-operator-77f648d694-jrgg9"
Jul 29 16:56:24.205232 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:56:24.205148    2570 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 29 16:56:24.205232 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:56:24.205185    2570 secret.go:281] references non-existent secret key: ca.crt
Jul 29 16:56:24.205232 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:56:24.205196    2570 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 29 16:56:24.205232 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:56:24.205210    2570 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-jrgg9: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 29 16:56:24.205405 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:56:24.205278    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/633c8dd1-fc22-44c7-9b77-3ec14af9d4c0-certificates podName:633c8dd1-fc22-44c7-9b77-3ec14af9d4c0 nodeName:}" failed. No retries permitted until 2026-07-29 16:56:24.705259403 +0000 UTC m=+313.194779053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/633c8dd1-fc22-44c7-9b77-3ec14af9d4c0-certificates") pod "keda-operator-77f648d694-jrgg9" (UID: "633c8dd1-fc22-44c7-9b77-3ec14af9d4c0") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 29 16:56:24.205780 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.205754    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/633c8dd1-fc22-44c7-9b77-3ec14af9d4c0-cabundle0\") pod \"keda-operator-77f648d694-jrgg9\" (UID: \"633c8dd1-fc22-44c7-9b77-3ec14af9d4c0\") " pod="openshift-keda/keda-operator-77f648d694-jrgg9"
Jul 29 16:56:24.217842 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.217804    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfndk\" (UniqueName: \"kubernetes.io/projected/633c8dd1-fc22-44c7-9b77-3ec14af9d4c0-kube-api-access-bfndk\") pod \"keda-operator-77f648d694-jrgg9\" (UID: \"633c8dd1-fc22-44c7-9b77-3ec14af9d4c0\") " pod="openshift-keda/keda-operator-77f648d694-jrgg9"
Jul 29 16:56:24.566761 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.566722    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"]
Jul 29 16:56:24.570072 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.570054    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"
Jul 29 16:56:24.572803 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.572779    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 29 16:56:24.580448 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.580424    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"]
Jul 29 16:56:24.710436 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.710390    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsn2v\" (UniqueName: \"kubernetes.io/projected/2dd9196c-8c6b-4c9e-820c-ecb24b6d1562-kube-api-access-hsn2v\") pod \"keda-metrics-apiserver-55557df8c-llmb5\" (UID: \"2dd9196c-8c6b-4c9e-820c-ecb24b6d1562\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"
Jul 29 16:56:24.710626 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.710447    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2dd9196c-8c6b-4c9e-820c-ecb24b6d1562-certificates\") pod \"keda-metrics-apiserver-55557df8c-llmb5\" (UID: \"2dd9196c-8c6b-4c9e-820c-ecb24b6d1562\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"
Jul 29 16:56:24.710626 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.710475    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/633c8dd1-fc22-44c7-9b77-3ec14af9d4c0-certificates\") pod \"keda-operator-77f648d694-jrgg9\" (UID: \"633c8dd1-fc22-44c7-9b77-3ec14af9d4c0\") " pod="openshift-keda/keda-operator-77f648d694-jrgg9"
Jul 29 16:56:24.710626 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.710568    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/2dd9196c-8c6b-4c9e-820c-ecb24b6d1562-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-llmb5\" (UID: \"2dd9196c-8c6b-4c9e-820c-ecb24b6d1562\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"
Jul 29 16:56:24.712938 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.712900    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/633c8dd1-fc22-44c7-9b77-3ec14af9d4c0-certificates\") pod \"keda-operator-77f648d694-jrgg9\" (UID: \"633c8dd1-fc22-44c7-9b77-3ec14af9d4c0\") " pod="openshift-keda/keda-operator-77f648d694-jrgg9"
Jul 29 16:56:24.811458 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.811416    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsn2v\" (UniqueName: \"kubernetes.io/projected/2dd9196c-8c6b-4c9e-820c-ecb24b6d1562-kube-api-access-hsn2v\") pod \"keda-metrics-apiserver-55557df8c-llmb5\" (UID: \"2dd9196c-8c6b-4c9e-820c-ecb24b6d1562\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"
Jul 29 16:56:24.811657 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.811477    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2dd9196c-8c6b-4c9e-820c-ecb24b6d1562-certificates\") pod \"keda-metrics-apiserver-55557df8c-llmb5\" (UID: \"2dd9196c-8c6b-4c9e-820c-ecb24b6d1562\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"
Jul 29 16:56:24.811657 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.811520    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/2dd9196c-8c6b-4c9e-820c-ecb24b6d1562-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-llmb5\" (UID: \"2dd9196c-8c6b-4c9e-820c-ecb24b6d1562\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"
Jul 29 16:56:24.811657 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:56:24.811643    2570 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 29 16:56:24.811831 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:56:24.811670    2570 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-llmb5: secret "keda-metrics-apiserver-certs" not found
Jul 29 16:56:24.811831 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:56:24.811737    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2dd9196c-8c6b-4c9e-820c-ecb24b6d1562-certificates podName:2dd9196c-8c6b-4c9e-820c-ecb24b6d1562 nodeName:}" failed. No retries permitted until 2026-07-29 16:56:25.311720554 +0000 UTC m=+313.801240188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2dd9196c-8c6b-4c9e-820c-ecb24b6d1562-certificates") pod "keda-metrics-apiserver-55557df8c-llmb5" (UID: "2dd9196c-8c6b-4c9e-820c-ecb24b6d1562") : secret "keda-metrics-apiserver-certs" not found
Jul 29 16:56:24.811979 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.811961    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/2dd9196c-8c6b-4c9e-820c-ecb24b6d1562-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-llmb5\" (UID: \"2dd9196c-8c6b-4c9e-820c-ecb24b6d1562\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"
Jul 29 16:56:24.822844 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.822770    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsn2v\" (UniqueName: \"kubernetes.io/projected/2dd9196c-8c6b-4c9e-820c-ecb24b6d1562-kube-api-access-hsn2v\") pod \"keda-metrics-apiserver-55557df8c-llmb5\" (UID: \"2dd9196c-8c6b-4c9e-820c-ecb24b6d1562\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"
Jul 29 16:56:24.930001 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:24.929960    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-jrgg9"
Jul 29 16:56:25.070468 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:25.070438    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-jrgg9"]
Jul 29 16:56:25.073039 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:56:25.072957    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633c8dd1_fc22_44c7_9b77_3ec14af9d4c0.slice/crio-e1e293cd504cc7dbe364a3837ba91f086017e2b2158b5228cfced2e00f8d2e6a WatchSource:0}: Error finding container e1e293cd504cc7dbe364a3837ba91f086017e2b2158b5228cfced2e00f8d2e6a: Status 404 returned error can't find the container with id e1e293cd504cc7dbe364a3837ba91f086017e2b2158b5228cfced2e00f8d2e6a
Jul 29 16:56:25.314394 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:25.314354    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2dd9196c-8c6b-4c9e-820c-ecb24b6d1562-certificates\") pod \"keda-metrics-apiserver-55557df8c-llmb5\" (UID: \"2dd9196c-8c6b-4c9e-820c-ecb24b6d1562\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"
Jul 29 16:56:25.316887 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:25.316866    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2dd9196c-8c6b-4c9e-820c-ecb24b6d1562-certificates\") pod \"keda-metrics-apiserver-55557df8c-llmb5\" (UID: \"2dd9196c-8c6b-4c9e-820c-ecb24b6d1562\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"
Jul 29 16:56:25.378594 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:25.378514    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-jrgg9" event={"ID":"633c8dd1-fc22-44c7-9b77-3ec14af9d4c0","Type":"ContainerStarted","Data":"e1e293cd504cc7dbe364a3837ba91f086017e2b2158b5228cfced2e00f8d2e6a"}
Jul 29 16:56:25.480519 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:25.480484    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"
Jul 29 16:56:25.628230 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:25.628204    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"]
Jul 29 16:56:25.630397 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:56:25.630330    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd9196c_8c6b_4c9e_820c_ecb24b6d1562.slice/crio-c980f7cb481939af41fa35126347f994472bce2d2c6a243c40f30416cbf25fbc WatchSource:0}: Error finding container c980f7cb481939af41fa35126347f994472bce2d2c6a243c40f30416cbf25fbc: Status 404 returned error can't find the container with id c980f7cb481939af41fa35126347f994472bce2d2c6a243c40f30416cbf25fbc
Jul 29 16:56:26.384989 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:26.384952    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5" event={"ID":"2dd9196c-8c6b-4c9e-820c-ecb24b6d1562","Type":"ContainerStarted","Data":"c980f7cb481939af41fa35126347f994472bce2d2c6a243c40f30416cbf25fbc"}
Jul 29 16:56:30.404981 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:30.404944    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5" event={"ID":"2dd9196c-8c6b-4c9e-820c-ecb24b6d1562","Type":"ContainerStarted","Data":"cc55a134e2c94a855d9f467539224c5bcf6ac75f76099c77f9652494c97bf211"}
Jul 29 16:56:30.405475 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:30.405008    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"
Jul 29 16:56:30.406403 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:30.406375    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-jrgg9" event={"ID":"633c8dd1-fc22-44c7-9b77-3ec14af9d4c0","Type":"ContainerStarted","Data":"b74eb15c99c450e351eb98d4132461b9b9000f994144026a7d5d47739f27356f"}
Jul 29 16:56:30.406493 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:30.406462    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-jrgg9"
Jul 29 16:56:30.428090 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:30.428040    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5" podStartSLOduration=2.618138452 podStartE2EDuration="6.428022588s" totalImagesPullingTime="3.809884136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:56:30.426403509 +0000 UTC m=+318.915923166" watchObservedRunningTime="2026-07-29 16:56:30.428022588 +0000 UTC m=+318.917542259"
Jul 29 16:56:30.457153 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:30.457065    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-jrgg9" podStartSLOduration=3.089692886 podStartE2EDuration="7.45704375s" totalImagesPullingTime="4.367350864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:56:30.45595496 +0000 UTC m=+318.945474617" watchObservedRunningTime="2026-07-29 16:56:30.45704375 +0000 UTC m=+318.946563409"
Jul 29 16:56:41.414982 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:41.414950    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-llmb5"
Jul 29 16:56:44.375813 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:44.375778    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-drr59"
Jul 29 16:56:51.412489 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:56:51.412458    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-jrgg9"
Jul 29 16:57:16.349221 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.349177    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"]
Jul 29 16:57:16.357991 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.357958    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"
Jul 29 16:57:16.361112 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.361080    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 16:57:16.361483 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.361460    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"]
Jul 29 16:57:16.361836 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.361808    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 16:57:16.361836 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.361852    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-tkjsw\""
Jul 29 16:57:16.409150 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.409101    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqhn9\" (UniqueName: \"kubernetes.io/projected/7602e6d0-3230-4759-a99a-be741f83544d-kube-api-access-nqhn9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf\" (UID: \"7602e6d0-3230-4759-a99a-be741f83544d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"
Jul 29 16:57:16.409356 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.409242    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7602e6d0-3230-4759-a99a-be741f83544d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf\" (UID: \"7602e6d0-3230-4759-a99a-be741f83544d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"
Jul 29 16:57:16.409356 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.409301    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7602e6d0-3230-4759-a99a-be741f83544d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf\" (UID: \"7602e6d0-3230-4759-a99a-be741f83544d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"
Jul 29 16:57:16.509924 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.509893    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7602e6d0-3230-4759-a99a-be741f83544d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf\" (UID: \"7602e6d0-3230-4759-a99a-be741f83544d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"
Jul 29 16:57:16.509924 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.509928    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqhn9\" (UniqueName: \"kubernetes.io/projected/7602e6d0-3230-4759-a99a-be741f83544d-kube-api-access-nqhn9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf\" (UID: \"7602e6d0-3230-4759-a99a-be741f83544d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"
Jul 29 16:57:16.510220 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.509997    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7602e6d0-3230-4759-a99a-be741f83544d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf\" (UID: \"7602e6d0-3230-4759-a99a-be741f83544d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"
Jul 29 16:57:16.510401 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.510377    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7602e6d0-3230-4759-a99a-be741f83544d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf\" (UID: \"7602e6d0-3230-4759-a99a-be741f83544d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"
Jul 29 16:57:16.510401 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.510391    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7602e6d0-3230-4759-a99a-be741f83544d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf\" (UID: \"7602e6d0-3230-4759-a99a-be741f83544d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"
Jul 29 16:57:16.518607 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.518573    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqhn9\" (UniqueName: \"kubernetes.io/projected/7602e6d0-3230-4759-a99a-be741f83544d-kube-api-access-nqhn9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf\" (UID: \"7602e6d0-3230-4759-a99a-be741f83544d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"
Jul 29 16:57:16.668264 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.668150    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"
Jul 29 16:57:16.805150 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:16.805119    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"]
Jul 29 16:57:16.808800 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:57:16.808765    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7602e6d0_3230_4759_a99a_be741f83544d.slice/crio-78d2e6ff6388cc1064f134603afbdab42edc4980777a19161bfd5fe25240b683 WatchSource:0}: Error finding container 78d2e6ff6388cc1064f134603afbdab42edc4980777a19161bfd5fe25240b683: Status 404 returned error can't find the container with id 78d2e6ff6388cc1064f134603afbdab42edc4980777a19161bfd5fe25240b683
Jul 29 16:57:17.607504 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:17.607466    2570 generic.go:356] "Generic (PLEG): container finished" podID="7602e6d0-3230-4759-a99a-be741f83544d" containerID="628c14ad8b0ab80e0d0e5c8b4607d4e6f4e7fd25063c08a3b7b64f3ef33a6014" exitCode=0
Jul 29 16:57:17.607904 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:17.607555    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf" event={"ID":"7602e6d0-3230-4759-a99a-be741f83544d","Type":"ContainerDied","Data":"628c14ad8b0ab80e0d0e5c8b4607d4e6f4e7fd25063c08a3b7b64f3ef33a6014"}
Jul 29 16:57:17.607904 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:17.607613    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf" event={"ID":"7602e6d0-3230-4759-a99a-be741f83544d","Type":"ContainerStarted","Data":"78d2e6ff6388cc1064f134603afbdab42edc4980777a19161bfd5fe25240b683"}
Jul 29 16:57:20.622760 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:20.622732    2570 generic.go:356] "Generic (PLEG): container finished" podID="7602e6d0-3230-4759-a99a-be741f83544d" containerID="16a083bf63dfb9a25d4605807aca1c8cbb213a6dd3c09e98de1d9702c3b56990" exitCode=0
Jul 29 16:57:20.623147 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:20.622785    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf" event={"ID":"7602e6d0-3230-4759-a99a-be741f83544d","Type":"ContainerDied","Data":"16a083bf63dfb9a25d4605807aca1c8cbb213a6dd3c09e98de1d9702c3b56990"}
Jul 29 16:57:21.629468 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:21.629436    2570 generic.go:356] "Generic (PLEG): container finished" podID="7602e6d0-3230-4759-a99a-be741f83544d" containerID="2b8bb09271f79dc9f56dae03dea879865884c4bd42c9449dc4215e407e432f27" exitCode=0
Jul 29 16:57:21.629955 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:21.629535    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf" event={"ID":"7602e6d0-3230-4759-a99a-be741f83544d","Type":"ContainerDied","Data":"2b8bb09271f79dc9f56dae03dea879865884c4bd42c9449dc4215e407e432f27"}
Jul 29 16:57:22.767414 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:22.767372    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"
Jul 29 16:57:22.865091 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:22.865053    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7602e6d0-3230-4759-a99a-be741f83544d-bundle\") pod \"7602e6d0-3230-4759-a99a-be741f83544d\" (UID: \"7602e6d0-3230-4759-a99a-be741f83544d\") "
Jul 29 16:57:22.865274 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:22.865106    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqhn9\" (UniqueName: \"kubernetes.io/projected/7602e6d0-3230-4759-a99a-be741f83544d-kube-api-access-nqhn9\") pod \"7602e6d0-3230-4759-a99a-be741f83544d\" (UID: \"7602e6d0-3230-4759-a99a-be741f83544d\") "
Jul 29 16:57:22.865274 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:22.865132    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7602e6d0-3230-4759-a99a-be741f83544d-util\") pod \"7602e6d0-3230-4759-a99a-be741f83544d\" (UID: \"7602e6d0-3230-4759-a99a-be741f83544d\") "
Jul 29 16:57:22.865834 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:22.865808    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7602e6d0-3230-4759-a99a-be741f83544d-bundle" (OuterVolumeSpecName: "bundle") pod "7602e6d0-3230-4759-a99a-be741f83544d" (UID: "7602e6d0-3230-4759-a99a-be741f83544d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:57:22.867179 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:22.867138    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7602e6d0-3230-4759-a99a-be741f83544d-kube-api-access-nqhn9" (OuterVolumeSpecName: "kube-api-access-nqhn9") pod "7602e6d0-3230-4759-a99a-be741f83544d" (UID: "7602e6d0-3230-4759-a99a-be741f83544d"). InnerVolumeSpecName "kube-api-access-nqhn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:57:22.869922 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:22.869899    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7602e6d0-3230-4759-a99a-be741f83544d-util" (OuterVolumeSpecName: "util") pod "7602e6d0-3230-4759-a99a-be741f83544d" (UID: "7602e6d0-3230-4759-a99a-be741f83544d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:57:22.966522 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:22.966440    2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7602e6d0-3230-4759-a99a-be741f83544d-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:57:22.966522 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:22.966471    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqhn9\" (UniqueName: \"kubernetes.io/projected/7602e6d0-3230-4759-a99a-be741f83544d-kube-api-access-nqhn9\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:57:22.966522 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:22.966481    2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7602e6d0-3230-4759-a99a-be741f83544d-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:57:23.639242 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:23.639207    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf" event={"ID":"7602e6d0-3230-4759-a99a-be741f83544d","Type":"ContainerDied","Data":"78d2e6ff6388cc1064f134603afbdab42edc4980777a19161bfd5fe25240b683"}
Jul 29 16:57:23.639242 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:23.639232    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540twwgf"
Jul 29 16:57:23.639454 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:23.639264    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78d2e6ff6388cc1064f134603afbdab42edc4980777a19161bfd5fe25240b683"
Jul 29 16:57:29.521331 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.521294    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8vg9b"]
Jul 29 16:57:29.521827 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.521728    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7602e6d0-3230-4759-a99a-be741f83544d" containerName="util"
Jul 29 16:57:29.521827 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.521745    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7602e6d0-3230-4759-a99a-be741f83544d" containerName="util"
Jul 29 16:57:29.521827 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.521756    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7602e6d0-3230-4759-a99a-be741f83544d" containerName="extract"
Jul 29 16:57:29.521827 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.521764    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7602e6d0-3230-4759-a99a-be741f83544d" containerName="extract"
Jul 29 16:57:29.521827 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.521786    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7602e6d0-3230-4759-a99a-be741f83544d" containerName="pull"
Jul 29 16:57:29.521827 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.521795    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7602e6d0-3230-4759-a99a-be741f83544d" containerName="pull"
Jul 29 16:57:29.522111 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.521867    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7602e6d0-3230-4759-a99a-be741f83544d" containerName="extract"
Jul 29 16:57:29.525756 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.525737    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8vg9b"
Jul 29 16:57:29.528525 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.528506    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 29 16:57:29.528658 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.528566    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 29 16:57:29.528719 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.528701    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-blss4\""
Jul 29 16:57:29.539797 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.539767    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8vg9b"]
Jul 29 16:57:29.622364 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.622330    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b72nd\" (UniqueName: \"kubernetes.io/projected/20229f39-31d1-4bc3-b7b7-e15860469ce7-kube-api-access-b72nd\") pod \"cert-manager-operator-controller-manager-55c444bf86-8vg9b\" (UID: \"20229f39-31d1-4bc3-b7b7-e15860469ce7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8vg9b"
Jul 29 16:57:29.622528 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.622394    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20229f39-31d1-4bc3-b7b7-e15860469ce7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8vg9b\" (UID: \"20229f39-31d1-4bc3-b7b7-e15860469ce7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8vg9b"
Jul 29 16:57:29.723526 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.723485    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20229f39-31d1-4bc3-b7b7-e15860469ce7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8vg9b\" (UID: \"20229f39-31d1-4bc3-b7b7-e15860469ce7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8vg9b"
Jul 29 16:57:29.723725 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.723587    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b72nd\" (UniqueName: \"kubernetes.io/projected/20229f39-31d1-4bc3-b7b7-e15860469ce7-kube-api-access-b72nd\") pod \"cert-manager-operator-controller-manager-55c444bf86-8vg9b\" (UID: \"20229f39-31d1-4bc3-b7b7-e15860469ce7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8vg9b"
Jul 29 16:57:29.723866 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.723844    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20229f39-31d1-4bc3-b7b7-e15860469ce7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8vg9b\" (UID: \"20229f39-31d1-4bc3-b7b7-e15860469ce7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8vg9b"
Jul 29 16:57:29.733109 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.733079    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b72nd\" (UniqueName: \"kubernetes.io/projected/20229f39-31d1-4bc3-b7b7-e15860469ce7-kube-api-access-b72nd\") pod \"cert-manager-operator-controller-manager-55c444bf86-8vg9b\" (UID: \"20229f39-31d1-4bc3-b7b7-e15860469ce7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8vg9b"
Jul 29 16:57:29.834581 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.834498    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8vg9b"
Jul 29 16:57:29.976244 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:29.976215    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8vg9b"]
Jul 29 16:57:29.978829 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:57:29.978799    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20229f39_31d1_4bc3_b7b7_e15860469ce7.slice/crio-efef750f862468f7db1f2f39fa66c361ee505ec8ed8f6279e5ac49779368be63 WatchSource:0}: Error finding container efef750f862468f7db1f2f39fa66c361ee505ec8ed8f6279e5ac49779368be63: Status 404 returned error can't find the container with id efef750f862468f7db1f2f39fa66c361ee505ec8ed8f6279e5ac49779368be63
Jul 29 16:57:30.669719 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:30.669678    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8vg9b" event={"ID":"20229f39-31d1-4bc3-b7b7-e15860469ce7","Type":"ContainerStarted","Data":"efef750f862468f7db1f2f39fa66c361ee505ec8ed8f6279e5ac49779368be63"}
Jul 29 16:57:32.685239 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:32.685201    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8vg9b" event={"ID":"20229f39-31d1-4bc3-b7b7-e15860469ce7","Type":"ContainerStarted","Data":"472bcdbb889b4ce4665276b21139341a592b166a7f212eac5d407c7713393307"}
Jul 29 16:57:32.718273 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:32.718220    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8vg9b" podStartSLOduration=1.935353275 podStartE2EDuration="3.718200932s" totalImagesPullingTime="1.782847657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:57:32.7175657 +0000 UTC m=+381.207085356" watchObservedRunningTime="2026-07-29 16:57:32.718200932 +0000 UTC m=+381.207720588"
Jul 29 16:57:38.793139 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:38.793094    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"]
Jul 29 16:57:38.801934 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:38.796987    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"
Jul 29 16:57:38.802500 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:38.802476    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-tkjsw\""
Jul 29 16:57:38.802634 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:38.802471    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 16:57:38.803071 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:38.803055    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 16:57:38.812511 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:38.812481    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"]
Jul 29 16:57:38.898343 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:38.898304    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b3d3b2-6c35-425f-ac12-72c7a170687a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n\" (UID: \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"
Jul 29 16:57:38.898666 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:38.898638    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtj5q\" (UniqueName: \"kubernetes.io/projected/c0b3d3b2-6c35-425f-ac12-72c7a170687a-kube-api-access-jtj5q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n\" (UID: \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"
Jul 29 16:57:38.898763 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:38.898718    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0b3d3b2-6c35-425f-ac12-72c7a170687a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n\" (UID: \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"
Jul 29 16:57:38.999587 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:38.999545    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0b3d3b2-6c35-425f-ac12-72c7a170687a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n\" (UID: \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"
Jul 29 16:57:38.999761 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:38.999599    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b3d3b2-6c35-425f-ac12-72c7a170687a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n\" (UID: \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"
Jul 29 16:57:38.999761 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:38.999641    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtj5q\" (UniqueName: \"kubernetes.io/projected/c0b3d3b2-6c35-425f-ac12-72c7a170687a-kube-api-access-jtj5q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n\" (UID: \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"
Jul 29 16:57:38.999917 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:38.999896    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0b3d3b2-6c35-425f-ac12-72c7a170687a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n\" (UID: \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"
Jul 29 16:57:38.999982 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:38.999963    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b3d3b2-6c35-425f-ac12-72c7a170687a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n\" (UID: \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"
Jul 29 16:57:39.008642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:39.008604    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtj5q\" (UniqueName: \"kubernetes.io/projected/c0b3d3b2-6c35-425f-ac12-72c7a170687a-kube-api-access-jtj5q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n\" (UID: \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"
Jul 29 16:57:39.111436 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:39.111342    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"
Jul 29 16:57:39.251206 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:39.251181    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"]
Jul 29 16:57:39.253707 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:57:39.253680    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b3d3b2_6c35_425f_ac12_72c7a170687a.slice/crio-1d9ea7bf3579a4d01d12e930b352692b33edf7689a6d4c2fb0ec599e90144254 WatchSource:0}: Error finding container 1d9ea7bf3579a4d01d12e930b352692b33edf7689a6d4c2fb0ec599e90144254: Status 404 returned error can't find the container with id 1d9ea7bf3579a4d01d12e930b352692b33edf7689a6d4c2fb0ec599e90144254
Jul 29 16:57:39.716794 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:39.716707    2570 generic.go:356] "Generic (PLEG): container finished" podID="c0b3d3b2-6c35-425f-ac12-72c7a170687a" containerID="769083fafd44ef2850169e8090bcad650323324aeff4a89cac5989ba37f4201d" exitCode=0
Jul 29 16:57:39.716794 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:39.716766    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n" event={"ID":"c0b3d3b2-6c35-425f-ac12-72c7a170687a","Type":"ContainerDied","Data":"769083fafd44ef2850169e8090bcad650323324aeff4a89cac5989ba37f4201d"}
Jul 29 16:57:39.716980 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:39.716803    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n" event={"ID":"c0b3d3b2-6c35-425f-ac12-72c7a170687a","Type":"ContainerStarted","Data":"1d9ea7bf3579a4d01d12e930b352692b33edf7689a6d4c2fb0ec599e90144254"}
Jul 29 16:57:42.734004 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:42.733970    2570 generic.go:356] "Generic (PLEG): container finished" podID="c0b3d3b2-6c35-425f-ac12-72c7a170687a" containerID="1052be3bb6a904d82b789324a7b606382c6af4347327c5e3718ecc0344d55db2" exitCode=0
Jul 29 16:57:42.734486 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:42.734054    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n" event={"ID":"c0b3d3b2-6c35-425f-ac12-72c7a170687a","Type":"ContainerDied","Data":"1052be3bb6a904d82b789324a7b606382c6af4347327c5e3718ecc0344d55db2"}
Jul 29 16:57:43.741504 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:43.741458    2570 generic.go:356] "Generic (PLEG): container finished" podID="c0b3d3b2-6c35-425f-ac12-72c7a170687a" containerID="5f3c0b28fd6aab1fa880694fec24612f662447b35d33b6b45b513903e5acff6a" exitCode=0
Jul 29 16:57:43.741917 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:43.741537    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n" event={"ID":"c0b3d3b2-6c35-425f-ac12-72c7a170687a","Type":"ContainerDied","Data":"5f3c0b28fd6aab1fa880694fec24612f662447b35d33b6b45b513903e5acff6a"}
Jul 29 16:57:44.685951 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:44.685914    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6rwvc"]
Jul 29 16:57:44.690031 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:44.690009    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6rwvc"
Jul 29 16:57:44.692520 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:44.692492    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 29 16:57:44.692520 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:44.692517    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 29 16:57:44.693428 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:44.693405    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-lvghh\""
Jul 29 16:57:44.702035 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:44.702003    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6rwvc"]
Jul 29 16:57:44.849517 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:44.849479    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/262ed140-8172-40c5-bc6a-fcdf9216eacb-bound-sa-token\") pod \"cert-manager-55b4d954d5-6rwvc\" (UID: \"262ed140-8172-40c5-bc6a-fcdf9216eacb\") " pod="cert-manager/cert-manager-55b4d954d5-6rwvc"
Jul 29 16:57:44.849969 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:44.849627    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlkdj\" (UniqueName: \"kubernetes.io/projected/262ed140-8172-40c5-bc6a-fcdf9216eacb-kube-api-access-vlkdj\") pod \"cert-manager-55b4d954d5-6rwvc\" (UID: \"262ed140-8172-40c5-bc6a-fcdf9216eacb\") " pod="cert-manager/cert-manager-55b4d954d5-6rwvc"
Jul 29 16:57:44.882883 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:44.882848    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"
Jul 29 16:57:44.951195 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:44.951074    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/262ed140-8172-40c5-bc6a-fcdf9216eacb-bound-sa-token\") pod \"cert-manager-55b4d954d5-6rwvc\" (UID: \"262ed140-8172-40c5-bc6a-fcdf9216eacb\") " pod="cert-manager/cert-manager-55b4d954d5-6rwvc"
Jul 29 16:57:44.951195 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:44.951148    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlkdj\" (UniqueName: \"kubernetes.io/projected/262ed140-8172-40c5-bc6a-fcdf9216eacb-kube-api-access-vlkdj\") pod \"cert-manager-55b4d954d5-6rwvc\" (UID: \"262ed140-8172-40c5-bc6a-fcdf9216eacb\") " pod="cert-manager/cert-manager-55b4d954d5-6rwvc"
Jul 29 16:57:44.961052 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:44.961022    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/262ed140-8172-40c5-bc6a-fcdf9216eacb-bound-sa-token\") pod \"cert-manager-55b4d954d5-6rwvc\" (UID: \"262ed140-8172-40c5-bc6a-fcdf9216eacb\") " pod="cert-manager/cert-manager-55b4d954d5-6rwvc"
Jul 29 16:57:44.961258 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:44.961101    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlkdj\" (UniqueName: \"kubernetes.io/projected/262ed140-8172-40c5-bc6a-fcdf9216eacb-kube-api-access-vlkdj\") pod \"cert-manager-55b4d954d5-6rwvc\" (UID: \"262ed140-8172-40c5-bc6a-fcdf9216eacb\") " pod="cert-manager/cert-manager-55b4d954d5-6rwvc"
Jul 29 16:57:45.010140 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.010102    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6rwvc"
Jul 29 16:57:45.052367 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.052325    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0b3d3b2-6c35-425f-ac12-72c7a170687a-util\") pod \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\" (UID: \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\") "
Jul 29 16:57:45.052543 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.052400    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtj5q\" (UniqueName: \"kubernetes.io/projected/c0b3d3b2-6c35-425f-ac12-72c7a170687a-kube-api-access-jtj5q\") pod \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\" (UID: \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\") "
Jul 29 16:57:45.052543 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.052460    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b3d3b2-6c35-425f-ac12-72c7a170687a-bundle\") pod \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\" (UID: \"c0b3d3b2-6c35-425f-ac12-72c7a170687a\") "
Jul 29 16:57:45.052877 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.052846    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b3d3b2-6c35-425f-ac12-72c7a170687a-bundle" (OuterVolumeSpecName: "bundle") pod "c0b3d3b2-6c35-425f-ac12-72c7a170687a" (UID: "c0b3d3b2-6c35-425f-ac12-72c7a170687a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:57:45.054806 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.054771    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b3d3b2-6c35-425f-ac12-72c7a170687a-kube-api-access-jtj5q" (OuterVolumeSpecName: "kube-api-access-jtj5q") pod "c0b3d3b2-6c35-425f-ac12-72c7a170687a" (UID: "c0b3d3b2-6c35-425f-ac12-72c7a170687a"). InnerVolumeSpecName "kube-api-access-jtj5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:57:45.057757 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.057701    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b3d3b2-6c35-425f-ac12-72c7a170687a-util" (OuterVolumeSpecName: "util") pod "c0b3d3b2-6c35-425f-ac12-72c7a170687a" (UID: "c0b3d3b2-6c35-425f-ac12-72c7a170687a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:57:45.153329 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.153298    2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0b3d3b2-6c35-425f-ac12-72c7a170687a-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:57:45.153329 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.153326    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtj5q\" (UniqueName: \"kubernetes.io/projected/c0b3d3b2-6c35-425f-ac12-72c7a170687a-kube-api-access-jtj5q\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:57:45.153329 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.153337    2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b3d3b2-6c35-425f-ac12-72c7a170687a-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:57:45.153983 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.153959    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6rwvc"]
Jul 29 16:57:45.156587 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:57:45.156550    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod262ed140_8172_40c5_bc6a_fcdf9216eacb.slice/crio-36992718cd843f07a8f1329fd864bbb22301562363dc5d18fd94cd77fb4a2651 WatchSource:0}: Error finding container 36992718cd843f07a8f1329fd864bbb22301562363dc5d18fd94cd77fb4a2651: Status 404 returned error can't find the container with id 36992718cd843f07a8f1329fd864bbb22301562363dc5d18fd94cd77fb4a2651
Jul 29 16:57:45.752558 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.752525    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n"
Jul 29 16:57:45.752712 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.752556    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffcm6n" event={"ID":"c0b3d3b2-6c35-425f-ac12-72c7a170687a","Type":"ContainerDied","Data":"1d9ea7bf3579a4d01d12e930b352692b33edf7689a6d4c2fb0ec599e90144254"}
Jul 29 16:57:45.752712 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.752614    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d9ea7bf3579a4d01d12e930b352692b33edf7689a6d4c2fb0ec599e90144254"
Jul 29 16:57:45.753646 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:45.753622    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6rwvc" event={"ID":"262ed140-8172-40c5-bc6a-fcdf9216eacb","Type":"ContainerStarted","Data":"36992718cd843f07a8f1329fd864bbb22301562363dc5d18fd94cd77fb4a2651"}
Jul 29 16:57:48.769677 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:48.769637    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6rwvc" event={"ID":"262ed140-8172-40c5-bc6a-fcdf9216eacb","Type":"ContainerStarted","Data":"99049c3521c3bcec6a8b7c8853e819fd9a92558bcb9363edd41adf54b65a376d"}
Jul 29 16:57:48.792836 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:57:48.792395    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-6rwvc" podStartSLOduration=2.124099084 podStartE2EDuration="4.792372896s" totalImagesPullingTime="2.668273812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:57:48.791878431 +0000 UTC m=+397.281398087" watchObservedRunningTime="2026-07-29 16:57:48.792372896 +0000 UTC m=+397.281892553"
Jul 29 16:58:05.902198 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.902147    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"]
Jul 29 16:58:05.902616 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.902489    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b3d3b2-6c35-425f-ac12-72c7a170687a" containerName="pull"
Jul 29 16:58:05.902616 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.902503    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b3d3b2-6c35-425f-ac12-72c7a170687a" containerName="pull"
Jul 29 16:58:05.902616 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.902512    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b3d3b2-6c35-425f-ac12-72c7a170687a" containerName="util"
Jul 29 16:58:05.902616 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.902520    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b3d3b2-6c35-425f-ac12-72c7a170687a" containerName="util"
Jul 29 16:58:05.902616 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.902526    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b3d3b2-6c35-425f-ac12-72c7a170687a" containerName="extract"
Jul 29 16:58:05.902616 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.902532    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b3d3b2-6c35-425f-ac12-72c7a170687a" containerName="extract"
Jul 29 16:58:05.902616 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.902585    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b3d3b2-6c35-425f-ac12-72c7a170687a" containerName="extract"
Jul 29 16:58:05.905427 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.905408    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:05.907849 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.907818    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 29 16:58:05.908023 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.907908    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 29 16:58:05.908105 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.908093    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 29 16:58:05.908184 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.908104    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 16:58:05.908184 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.908151    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ccb4n\""
Jul 29 16:58:05.919895 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.919860    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"]
Jul 29 16:58:05.929363 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.929325    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/167a4d72-f5d1-4be5-895d-1f96bea939c4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:05.929553 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.929383    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:05.929553 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.929448    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:05.929674 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.929562    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:05.929674 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.929602    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:05.929838 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.929817    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:05.929940 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:05.929913    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68m7b\" (UniqueName: \"kubernetes.io/projected/167a4d72-f5d1-4be5-895d-1f96bea939c4-kube-api-access-68m7b\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.030564 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.030529    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.030741 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.030578    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68m7b\" (UniqueName: \"kubernetes.io/projected/167a4d72-f5d1-4be5-895d-1f96bea939c4-kube-api-access-68m7b\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.030741 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.030604    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/167a4d72-f5d1-4be5-895d-1f96bea939c4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.030741 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.030633    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.030741 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.030665    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.030741 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.030716    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.030741 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.030739    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.031385 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.031352    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.033201 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.033155    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.033332 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.033263    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.033332 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.033266    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/167a4d72-f5d1-4be5-895d-1f96bea939c4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.033332 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.033319    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.039385 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.039349    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.039539 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.039423    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68m7b\" (UniqueName: \"kubernetes.io/projected/167a4d72-f5d1-4be5-895d-1f96bea939c4-kube-api-access-68m7b\") pod \"istiod-openshift-gateway-7b7995d5cd-hxqvm\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.215241 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.215129    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:06.368978 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.368954    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"]
Jul 29 16:58:06.371327 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:58:06.371296    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167a4d72_f5d1_4be5_895d_1f96bea939c4.slice/crio-48ef05df30c269179a44ccd42c3c44f8c22c944f37f459b820439cbc23b2c367 WatchSource:0}: Error finding container 48ef05df30c269179a44ccd42c3c44f8c22c944f37f459b820439cbc23b2c367: Status 404 returned error can't find the container with id 48ef05df30c269179a44ccd42c3c44f8c22c944f37f459b820439cbc23b2c367
Jul 29 16:58:06.861726 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:06.861666    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm" event={"ID":"167a4d72-f5d1-4be5-895d-1f96bea939c4","Type":"ContainerStarted","Data":"48ef05df30c269179a44ccd42c3c44f8c22c944f37f459b820439cbc23b2c367"}
Jul 29 16:58:09.334782 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:09.334746    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 29 16:58:09.335147 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:09.334825    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 29 16:58:09.883182 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:09.883115    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm" event={"ID":"167a4d72-f5d1-4be5-895d-1f96bea939c4","Type":"ContainerStarted","Data":"44e02346bc5b0b9f841c4dc7f08d0b76a8afb42c0adaa5d271f568ebe9cdab16"}
Jul 29 16:58:09.883533 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:09.883509    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:09.907519 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:09.907448    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm" podStartSLOduration=1.9462487309999998 podStartE2EDuration="4.907430877s" totalImagesPullingTime="2.961182146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:58:09.904740123 +0000 UTC m=+418.394259800" watchObservedRunningTime="2026-07-29 16:58:09.907430877 +0000 UTC m=+418.396950533"
Jul 29 16:58:10.889413 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:10.889385    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 16:58:20.128260 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:20.128224    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5859d65dc9-4hlnh"]
Jul 29 16:58:22.689858 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.689825    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"]
Jul 29 16:58:22.694482 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.694461    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"
Jul 29 16:58:22.697205 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.697155    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 16:58:22.697366 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.697234    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 16:58:22.697366 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.697348    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-tkjsw\""
Jul 29 16:58:22.704879 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.704854    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"]
Jul 29 16:58:22.778352 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.778318    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pznkh\" (UniqueName: \"kubernetes.io/projected/795ce7ec-2f54-4995-95c1-d41575a19860-kube-api-access-pznkh\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr\" (UID: \"795ce7ec-2f54-4995-95c1-d41575a19860\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"
Jul 29 16:58:22.778529 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.778396    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795ce7ec-2f54-4995-95c1-d41575a19860-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr\" (UID: \"795ce7ec-2f54-4995-95c1-d41575a19860\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"
Jul 29 16:58:22.778529 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.778436    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795ce7ec-2f54-4995-95c1-d41575a19860-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr\" (UID: \"795ce7ec-2f54-4995-95c1-d41575a19860\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"
Jul 29 16:58:22.792604 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.792572    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"]
Jul 29 16:58:22.796075 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.796055    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"
Jul 29 16:58:22.806715 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.806688    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"]
Jul 29 16:58:22.879873 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.879833    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795ce7ec-2f54-4995-95c1-d41575a19860-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr\" (UID: \"795ce7ec-2f54-4995-95c1-d41575a19860\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"
Jul 29 16:58:22.880057 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.879884    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795ce7ec-2f54-4995-95c1-d41575a19860-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr\" (UID: \"795ce7ec-2f54-4995-95c1-d41575a19860\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"
Jul 29 16:58:22.880057 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.879915    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f759f258-3c4f-4d0d-8a56-f044033acf22-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx\" (UID: \"f759f258-3c4f-4d0d-8a56-f044033acf22\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"
Jul 29 16:58:22.880057 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.879980    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pznkh\" (UniqueName: \"kubernetes.io/projected/795ce7ec-2f54-4995-95c1-d41575a19860-kube-api-access-pznkh\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr\" (UID: \"795ce7ec-2f54-4995-95c1-d41575a19860\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"
Jul 29 16:58:22.880057 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.880024    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f759f258-3c4f-4d0d-8a56-f044033acf22-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx\" (UID: \"f759f258-3c4f-4d0d-8a56-f044033acf22\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"
Jul 29 16:58:22.880318 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.880065    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c75l\" (UniqueName: \"kubernetes.io/projected/f759f258-3c4f-4d0d-8a56-f044033acf22-kube-api-access-5c75l\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx\" (UID: \"f759f258-3c4f-4d0d-8a56-f044033acf22\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"
Jul 29 16:58:22.880318 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.880257    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795ce7ec-2f54-4995-95c1-d41575a19860-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr\" (UID: \"795ce7ec-2f54-4995-95c1-d41575a19860\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"
Jul 29 16:58:22.880389 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.880313    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795ce7ec-2f54-4995-95c1-d41575a19860-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr\" (UID: \"795ce7ec-2f54-4995-95c1-d41575a19860\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"
Jul 29 16:58:22.888537 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.888506    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pznkh\" (UniqueName: \"kubernetes.io/projected/795ce7ec-2f54-4995-95c1-d41575a19860-kube-api-access-pznkh\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr\" (UID: \"795ce7ec-2f54-4995-95c1-d41575a19860\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"
Jul 29 16:58:22.893617 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.893587    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"]
Jul 29 16:58:22.897346 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.897321    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"
Jul 29 16:58:22.909816 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.909787    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"]
Jul 29 16:58:22.980584 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.980483    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5\" (UID: \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"
Jul 29 16:58:22.980584 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.980530    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f759f258-3c4f-4d0d-8a56-f044033acf22-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx\" (UID: \"f759f258-3c4f-4d0d-8a56-f044033acf22\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"
Jul 29 16:58:22.980584 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.980560    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5\" (UID: \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"
Jul 29 16:58:22.980868 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.980592    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f759f258-3c4f-4d0d-8a56-f044033acf22-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx\" (UID: \"f759f258-3c4f-4d0d-8a56-f044033acf22\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"
Jul 29 16:58:22.980868 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.980637    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c75l\" (UniqueName: \"kubernetes.io/projected/f759f258-3c4f-4d0d-8a56-f044033acf22-kube-api-access-5c75l\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx\" (UID: \"f759f258-3c4f-4d0d-8a56-f044033acf22\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"
Jul 29 16:58:22.980868 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.980668    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfjvc\" (UniqueName: \"kubernetes.io/projected/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-kube-api-access-qfjvc\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5\" (UID: \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"
Jul 29 16:58:22.981047 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.980924    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f759f258-3c4f-4d0d-8a56-f044033acf22-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx\" (UID: \"f759f258-3c4f-4d0d-8a56-f044033acf22\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"
Jul 29 16:58:22.981047 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.980993    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f759f258-3c4f-4d0d-8a56-f044033acf22-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx\" (UID: \"f759f258-3c4f-4d0d-8a56-f044033acf22\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"
Jul 29 16:58:22.992274 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.992243    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c75l\" (UniqueName: \"kubernetes.io/projected/f759f258-3c4f-4d0d-8a56-f044033acf22-kube-api-access-5c75l\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx\" (UID: \"f759f258-3c4f-4d0d-8a56-f044033acf22\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"
Jul 29 16:58:22.993108 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.993079    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"]
Jul 29 16:58:22.996689 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:22.996665    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"
Jul 29 16:58:23.004701 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.004647    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"
Jul 29 16:58:23.008487 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.008459    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"]
Jul 29 16:58:23.083195 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.082647    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5\" (UID: \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"
Jul 29 16:58:23.093748 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.083074    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5\" (UID: \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"
Jul 29 16:58:23.093999 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.093922    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a9e361-fc9a-4b67-93ab-49cad41d1d72-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw\" (UID: \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"
Jul 29 16:58:23.094240 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.094151    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxmc9\" (UniqueName: \"kubernetes.io/projected/09a9e361-fc9a-4b67-93ab-49cad41d1d72-kube-api-access-bxmc9\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw\" (UID: \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"
Jul 29 16:58:23.094330 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.094304    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfjvc\" (UniqueName: \"kubernetes.io/projected/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-kube-api-access-qfjvc\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5\" (UID: \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"
Jul 29 16:58:23.094769 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.094381    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a9e361-fc9a-4b67-93ab-49cad41d1d72-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw\" (UID: \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"
Jul 29 16:58:23.094769 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.094444    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5\" (UID: \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"
Jul 29 16:58:23.094930 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.094838    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5\" (UID: \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"
Jul 29 16:58:23.103351 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.103239    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfjvc\" (UniqueName: \"kubernetes.io/projected/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-kube-api-access-qfjvc\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5\" (UID: \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"
Jul 29 16:58:23.105961 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.105935    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"
Jul 29 16:58:23.182582 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.182438    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"]
Jul 29 16:58:23.185342 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:58:23.185317    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795ce7ec_2f54_4995_95c1_d41575a19860.slice/crio-63e15b73d88bb2b46849d928065401192073f9f0f1c47e12d0b19f48004863a8 WatchSource:0}: Error finding container 63e15b73d88bb2b46849d928065401192073f9f0f1c47e12d0b19f48004863a8: Status 404 returned error can't find the container with id 63e15b73d88bb2b46849d928065401192073f9f0f1c47e12d0b19f48004863a8
Jul 29 16:58:23.195492 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.195470    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a9e361-fc9a-4b67-93ab-49cad41d1d72-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw\" (UID: \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"
Jul 29 16:58:23.195584 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.195502    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxmc9\" (UniqueName: \"kubernetes.io/projected/09a9e361-fc9a-4b67-93ab-49cad41d1d72-kube-api-access-bxmc9\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw\" (UID: \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"
Jul 29 16:58:23.195584 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.195546    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a9e361-fc9a-4b67-93ab-49cad41d1d72-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw\" (UID: \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"
Jul 29 16:58:23.195893 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.195870    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a9e361-fc9a-4b67-93ab-49cad41d1d72-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw\" (UID: \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"
Jul 29 16:58:23.195893 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.195885    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a9e361-fc9a-4b67-93ab-49cad41d1d72-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw\" (UID: \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"
Jul 29 16:58:23.204889 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.204858    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxmc9\" (UniqueName: \"kubernetes.io/projected/09a9e361-fc9a-4b67-93ab-49cad41d1d72-kube-api-access-bxmc9\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw\" (UID: \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"
Jul 29 16:58:23.206559 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.206518    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"
Jul 29 16:58:23.269464 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.269381    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"]
Jul 29 16:58:23.278041 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:58:23.275665    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf759f258_3c4f_4d0d_8a56_f044033acf22.slice/crio-af2de92c80b4da39c38bb2f6014b201f8beee86de5844626f6f6321f0b05ef4f WatchSource:0}: Error finding container af2de92c80b4da39c38bb2f6014b201f8beee86de5844626f6f6321f0b05ef4f: Status 404 returned error can't find the container with id af2de92c80b4da39c38bb2f6014b201f8beee86de5844626f6f6321f0b05ef4f
Jul 29 16:58:23.315501 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.315266    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"
Jul 29 16:58:23.372879 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.372268    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"]
Jul 29 16:58:23.374876 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:58:23.374850    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fb22be_f19b_4d9c_ade9_e7e8e22b06c5.slice/crio-c796d7ae8d217c5481d204772bdd36219272db1b9421af0611a7c503ebe5eaba WatchSource:0}: Error finding container c796d7ae8d217c5481d204772bdd36219272db1b9421af0611a7c503ebe5eaba: Status 404 returned error can't find the container with id c796d7ae8d217c5481d204772bdd36219272db1b9421af0611a7c503ebe5eaba
Jul 29 16:58:23.476743 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.476713    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"]
Jul 29 16:58:23.486264 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:58:23.486182    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a9e361_fc9a_4b67_93ab_49cad41d1d72.slice/crio-4de97041f5222af3dda5327251b84e291e9db69a8195aa2525af6f241da71762 WatchSource:0}: Error finding container 4de97041f5222af3dda5327251b84e291e9db69a8195aa2525af6f241da71762: Status 404 returned error can't find the container with id 4de97041f5222af3dda5327251b84e291e9db69a8195aa2525af6f241da71762
Jul 29 16:58:23.954233 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.954202    2570 generic.go:356] "Generic (PLEG): container finished" podID="f759f258-3c4f-4d0d-8a56-f044033acf22" containerID="88098688f2cfae833051bfa7911984c4e18b7360efee71d71ca91c47b1af15fa" exitCode=0
Jul 29 16:58:23.954697 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.954285    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx" event={"ID":"f759f258-3c4f-4d0d-8a56-f044033acf22","Type":"ContainerDied","Data":"88098688f2cfae833051bfa7911984c4e18b7360efee71d71ca91c47b1af15fa"}
Jul 29 16:58:23.954697 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.954337    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx" event={"ID":"f759f258-3c4f-4d0d-8a56-f044033acf22","Type":"ContainerStarted","Data":"af2de92c80b4da39c38bb2f6014b201f8beee86de5844626f6f6321f0b05ef4f"}
Jul 29 16:58:23.955904 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.955880    2570 generic.go:356] "Generic (PLEG): container finished" podID="b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5" containerID="5fe22232c46df0e4f764399e4eba6c3381d2a2ba7439d2eabce370ee686ed2f2" exitCode=0
Jul 29 16:58:23.955996 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.955965    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5" event={"ID":"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5","Type":"ContainerDied","Data":"5fe22232c46df0e4f764399e4eba6c3381d2a2ba7439d2eabce370ee686ed2f2"}
Jul 29 16:58:23.956060 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.956032    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5" event={"ID":"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5","Type":"ContainerStarted","Data":"c796d7ae8d217c5481d204772bdd36219272db1b9421af0611a7c503ebe5eaba"}
Jul 29 16:58:23.957621 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.957597    2570 generic.go:356] "Generic (PLEG): container finished" podID="795ce7ec-2f54-4995-95c1-d41575a19860" containerID="59b5b2f53a57bd8cf01e7b8649a723a3aecc963af7d3e3f48b2d2260112557af" exitCode=0
Jul 29 16:58:23.957718 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.957689    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr" event={"ID":"795ce7ec-2f54-4995-95c1-d41575a19860","Type":"ContainerDied","Data":"59b5b2f53a57bd8cf01e7b8649a723a3aecc963af7d3e3f48b2d2260112557af"}
Jul 29 16:58:23.957774 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.957740    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr" event={"ID":"795ce7ec-2f54-4995-95c1-d41575a19860","Type":"ContainerStarted","Data":"63e15b73d88bb2b46849d928065401192073f9f0f1c47e12d0b19f48004863a8"}
Jul 29 16:58:23.959696 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.959675    2570 generic.go:356] "Generic (PLEG): container finished" podID="09a9e361-fc9a-4b67-93ab-49cad41d1d72" containerID="f444ee78178877b7b064be1cdf9ce11f7af8236e1dcd835c5f5a3374551027c6" exitCode=0
Jul 29 16:58:23.959770 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.959722    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw" event={"ID":"09a9e361-fc9a-4b67-93ab-49cad41d1d72","Type":"ContainerDied","Data":"f444ee78178877b7b064be1cdf9ce11f7af8236e1dcd835c5f5a3374551027c6"}
Jul 29 16:58:23.959770 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:23.959757    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw" event={"ID":"09a9e361-fc9a-4b67-93ab-49cad41d1d72","Type":"ContainerStarted","Data":"4de97041f5222af3dda5327251b84e291e9db69a8195aa2525af6f241da71762"}
Jul 29 16:58:24.967709 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:24.967614    2570 generic.go:356] "Generic (PLEG): container finished" podID="f759f258-3c4f-4d0d-8a56-f044033acf22" containerID="c002f07b29aafa7dd9d4018ea8c22b1449d2ec85c6b659ec75e160e6994a72e6" exitCode=0
Jul 29 16:58:24.967709 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:24.967689    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx" event={"ID":"f759f258-3c4f-4d0d-8a56-f044033acf22","Type":"ContainerDied","Data":"c002f07b29aafa7dd9d4018ea8c22b1449d2ec85c6b659ec75e160e6994a72e6"}
Jul 29 16:58:25.975406 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:25.975355    2570 generic.go:356] "Generic (PLEG): container finished" podID="f759f258-3c4f-4d0d-8a56-f044033acf22" containerID="19ac1543dcbb36da1fa7a2d5abeacb6fca207df328cdd8e4f1679ee1845de0a0" exitCode=0
Jul 29 16:58:25.975801 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:25.975445    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx" event={"ID":"f759f258-3c4f-4d0d-8a56-f044033acf22","Type":"ContainerDied","Data":"19ac1543dcbb36da1fa7a2d5abeacb6fca207df328cdd8e4f1679ee1845de0a0"}
Jul 29 16:58:25.977331 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:25.977298    2570 generic.go:356] "Generic (PLEG): container finished" podID="b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5" containerID="8088764bb3a581389dae35cc35f027dc0296a540e91e30fd7ba6fcd672be62c1" exitCode=0
Jul 29 16:58:25.977523 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:25.977384    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5" event={"ID":"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5","Type":"ContainerDied","Data":"8088764bb3a581389dae35cc35f027dc0296a540e91e30fd7ba6fcd672be62c1"}
Jul 29 16:58:25.979305 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:25.979277    2570 generic.go:356] "Generic (PLEG): container finished" podID="795ce7ec-2f54-4995-95c1-d41575a19860" containerID="e32f077c9dde9c66dd9757171eaee99b5f7847725ba7db3fb25e4b9d4310c31f" exitCode=0
Jul 29 16:58:25.979421 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:25.979362    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr" event={"ID":"795ce7ec-2f54-4995-95c1-d41575a19860","Type":"ContainerDied","Data":"e32f077c9dde9c66dd9757171eaee99b5f7847725ba7db3fb25e4b9d4310c31f"}
Jul 29 16:58:26.986145 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:26.986107    2570 generic.go:356] "Generic (PLEG): container finished" podID="b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5" containerID="9f6c1dd9b5e2f39b50556b4242c24278438d87e3ae98c65789feaa28e57c221e" exitCode=0
Jul 29 16:58:26.986582 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:26.986192    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5" event={"ID":"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5","Type":"ContainerDied","Data":"9f6c1dd9b5e2f39b50556b4242c24278438d87e3ae98c65789feaa28e57c221e"}
Jul 29 16:58:26.988306 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:26.988281    2570 generic.go:356] "Generic (PLEG): container finished" podID="795ce7ec-2f54-4995-95c1-d41575a19860" containerID="40688621ebf50bd9e6c32eb35cb92b5779efd84a8eb1bb90ba5d9d97944e2f35" exitCode=0
Jul 29 16:58:26.988406 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:26.988361    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr" event={"ID":"795ce7ec-2f54-4995-95c1-d41575a19860","Type":"ContainerDied","Data":"40688621ebf50bd9e6c32eb35cb92b5779efd84a8eb1bb90ba5d9d97944e2f35"}
Jul 29 16:58:26.989891 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:26.989871    2570 generic.go:356] "Generic (PLEG): container finished" podID="09a9e361-fc9a-4b67-93ab-49cad41d1d72" containerID="e9654bcc00c397901b77c415057e65851045cdcc8daf05a8c2157c5c0bea1f15" exitCode=0
Jul 29 16:58:26.989982 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:26.989926    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw" event={"ID":"09a9e361-fc9a-4b67-93ab-49cad41d1d72","Type":"ContainerDied","Data":"e9654bcc00c397901b77c415057e65851045cdcc8daf05a8c2157c5c0bea1f15"}
Jul 29 16:58:27.137274 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.137249    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"
Jul 29 16:58:27.233850 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.233819    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f759f258-3c4f-4d0d-8a56-f044033acf22-bundle\") pod \"f759f258-3c4f-4d0d-8a56-f044033acf22\" (UID: \"f759f258-3c4f-4d0d-8a56-f044033acf22\") "
Jul 29 16:58:27.234008 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.233867    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c75l\" (UniqueName: \"kubernetes.io/projected/f759f258-3c4f-4d0d-8a56-f044033acf22-kube-api-access-5c75l\") pod \"f759f258-3c4f-4d0d-8a56-f044033acf22\" (UID: \"f759f258-3c4f-4d0d-8a56-f044033acf22\") "
Jul 29 16:58:27.234008 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.233913    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f759f258-3c4f-4d0d-8a56-f044033acf22-util\") pod \"f759f258-3c4f-4d0d-8a56-f044033acf22\" (UID: \"f759f258-3c4f-4d0d-8a56-f044033acf22\") "
Jul 29 16:58:27.234437 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.234406    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f759f258-3c4f-4d0d-8a56-f044033acf22-bundle" (OuterVolumeSpecName: "bundle") pod "f759f258-3c4f-4d0d-8a56-f044033acf22" (UID: "f759f258-3c4f-4d0d-8a56-f044033acf22"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:58:27.236106 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.236081    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f759f258-3c4f-4d0d-8a56-f044033acf22-kube-api-access-5c75l" (OuterVolumeSpecName: "kube-api-access-5c75l") pod "f759f258-3c4f-4d0d-8a56-f044033acf22" (UID: "f759f258-3c4f-4d0d-8a56-f044033acf22"). InnerVolumeSpecName "kube-api-access-5c75l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:58:27.239193 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.239110    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f759f258-3c4f-4d0d-8a56-f044033acf22-util" (OuterVolumeSpecName: "util") pod "f759f258-3c4f-4d0d-8a56-f044033acf22" (UID: "f759f258-3c4f-4d0d-8a56-f044033acf22"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:58:27.335444 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.335408    2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f759f258-3c4f-4d0d-8a56-f044033acf22-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:27.335444 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.335438    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c75l\" (UniqueName: \"kubernetes.io/projected/f759f258-3c4f-4d0d-8a56-f044033acf22-kube-api-access-5c75l\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:27.335444 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.335447    2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f759f258-3c4f-4d0d-8a56-f044033acf22-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:27.996866 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.996832    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx"
Jul 29 16:58:27.997360 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.996835    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0z97cx" event={"ID":"f759f258-3c4f-4d0d-8a56-f044033acf22","Type":"ContainerDied","Data":"af2de92c80b4da39c38bb2f6014b201f8beee86de5844626f6f6321f0b05ef4f"}
Jul 29 16:58:27.997360 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.996938    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af2de92c80b4da39c38bb2f6014b201f8beee86de5844626f6f6321f0b05ef4f"
Jul 29 16:58:27.999063 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.999035    2570 generic.go:356] "Generic (PLEG): container finished" podID="09a9e361-fc9a-4b67-93ab-49cad41d1d72" containerID="b4c184b4f4b4828f9728fa2293ce94f212a6c672d2c237f475043a2ae44001a4" exitCode=0
Jul 29 16:58:27.999213 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:27.999126    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw" event={"ID":"09a9e361-fc9a-4b67-93ab-49cad41d1d72","Type":"ContainerDied","Data":"b4c184b4f4b4828f9728fa2293ce94f212a6c672d2c237f475043a2ae44001a4"}
Jul 29 16:58:28.164577 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.164549    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"
Jul 29 16:58:28.166569 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.166550    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"
Jul 29 16:58:28.244293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.244198    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795ce7ec-2f54-4995-95c1-d41575a19860-bundle\") pod \"795ce7ec-2f54-4995-95c1-d41575a19860\" (UID: \"795ce7ec-2f54-4995-95c1-d41575a19860\") "
Jul 29 16:58:28.244293 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.244253    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-util\") pod \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\" (UID: \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\") "
Jul 29 16:58:28.244476 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.244302    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfjvc\" (UniqueName: \"kubernetes.io/projected/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-kube-api-access-qfjvc\") pod \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\" (UID: \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\") "
Jul 29 16:58:28.244476 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.244368    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pznkh\" (UniqueName: \"kubernetes.io/projected/795ce7ec-2f54-4995-95c1-d41575a19860-kube-api-access-pznkh\") pod \"795ce7ec-2f54-4995-95c1-d41575a19860\" (UID: \"795ce7ec-2f54-4995-95c1-d41575a19860\") "
Jul 29 16:58:28.244476 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.244403    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-bundle\") pod \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\" (UID: \"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5\") "
Jul 29 16:58:28.244476 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.244455    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795ce7ec-2f54-4995-95c1-d41575a19860-util\") pod \"795ce7ec-2f54-4995-95c1-d41575a19860\" (UID: \"795ce7ec-2f54-4995-95c1-d41575a19860\") "
Jul 29 16:58:28.245052 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.245003    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-bundle" (OuterVolumeSpecName: "bundle") pod "b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5" (UID: "b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:58:28.245220 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.245085    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795ce7ec-2f54-4995-95c1-d41575a19860-bundle" (OuterVolumeSpecName: "bundle") pod "795ce7ec-2f54-4995-95c1-d41575a19860" (UID: "795ce7ec-2f54-4995-95c1-d41575a19860"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:58:28.246579 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.246553    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-kube-api-access-qfjvc" (OuterVolumeSpecName: "kube-api-access-qfjvc") pod "b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5" (UID: "b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5"). InnerVolumeSpecName "kube-api-access-qfjvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:58:28.246844 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.246814    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795ce7ec-2f54-4995-95c1-d41575a19860-kube-api-access-pznkh" (OuterVolumeSpecName: "kube-api-access-pznkh") pod "795ce7ec-2f54-4995-95c1-d41575a19860" (UID: "795ce7ec-2f54-4995-95c1-d41575a19860"). InnerVolumeSpecName "kube-api-access-pznkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:58:28.251072 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.251045    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-util" (OuterVolumeSpecName: "util") pod "b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5" (UID: "b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:58:28.252948 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.252914    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795ce7ec-2f54-4995-95c1-d41575a19860-util" (OuterVolumeSpecName: "util") pod "795ce7ec-2f54-4995-95c1-d41575a19860" (UID: "795ce7ec-2f54-4995-95c1-d41575a19860"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:58:28.345219 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.345157    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfjvc\" (UniqueName: \"kubernetes.io/projected/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-kube-api-access-qfjvc\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:28.345219 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.345215    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pznkh\" (UniqueName: \"kubernetes.io/projected/795ce7ec-2f54-4995-95c1-d41575a19860-kube-api-access-pznkh\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:28.345219 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.345226    2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:28.345468 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.345236    2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795ce7ec-2f54-4995-95c1-d41575a19860-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:28.345468 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.345244    2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795ce7ec-2f54-4995-95c1-d41575a19860-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:28.345468 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:28.345252    2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:29.005724 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.005692    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5" event={"ID":"b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5","Type":"ContainerDied","Data":"c796d7ae8d217c5481d204772bdd36219272db1b9421af0611a7c503ebe5eaba"}
Jul 29 16:58:29.006188 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.005739    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c796d7ae8d217c5481d204772bdd36219272db1b9421af0611a7c503ebe5eaba"
Jul 29 16:58:29.006188 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.005703    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cnvcd5"
Jul 29 16:58:29.007645 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.007625    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr"
Jul 29 16:58:29.007745 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.007659    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b2vkr" event={"ID":"795ce7ec-2f54-4995-95c1-d41575a19860","Type":"ContainerDied","Data":"63e15b73d88bb2b46849d928065401192073f9f0f1c47e12d0b19f48004863a8"}
Jul 29 16:58:29.007745 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.007705    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63e15b73d88bb2b46849d928065401192073f9f0f1c47e12d0b19f48004863a8"
Jul 29 16:58:29.151543 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.151517    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"
Jul 29 16:58:29.252889 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.252850    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a9e361-fc9a-4b67-93ab-49cad41d1d72-bundle\") pod \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\" (UID: \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\") "
Jul 29 16:58:29.253060 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.252899    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a9e361-fc9a-4b67-93ab-49cad41d1d72-util\") pod \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\" (UID: \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\") "
Jul 29 16:58:29.253060 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.252938    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxmc9\" (UniqueName: \"kubernetes.io/projected/09a9e361-fc9a-4b67-93ab-49cad41d1d72-kube-api-access-bxmc9\") pod \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\" (UID: \"09a9e361-fc9a-4b67-93ab-49cad41d1d72\") "
Jul 29 16:58:29.253408 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.253382    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a9e361-fc9a-4b67-93ab-49cad41d1d72-bundle" (OuterVolumeSpecName: "bundle") pod "09a9e361-fc9a-4b67-93ab-49cad41d1d72" (UID: "09a9e361-fc9a-4b67-93ab-49cad41d1d72"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:58:29.255115 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.255092    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a9e361-fc9a-4b67-93ab-49cad41d1d72-kube-api-access-bxmc9" (OuterVolumeSpecName: "kube-api-access-bxmc9") pod "09a9e361-fc9a-4b67-93ab-49cad41d1d72" (UID: "09a9e361-fc9a-4b67-93ab-49cad41d1d72"). InnerVolumeSpecName "kube-api-access-bxmc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:58:29.259602 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.259538    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a9e361-fc9a-4b67-93ab-49cad41d1d72-util" (OuterVolumeSpecName: "util") pod "09a9e361-fc9a-4b67-93ab-49cad41d1d72" (UID: "09a9e361-fc9a-4b67-93ab-49cad41d1d72"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:58:29.354405 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.354359    2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a9e361-fc9a-4b67-93ab-49cad41d1d72-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:29.354405 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.354396    2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a9e361-fc9a-4b67-93ab-49cad41d1d72-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:29.354405 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:29.354409    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxmc9\" (UniqueName: \"kubernetes.io/projected/09a9e361-fc9a-4b67-93ab-49cad41d1d72-kube-api-access-bxmc9\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:30.015649 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:30.015615    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw"
Jul 29 16:58:30.016149 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:30.015619    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhxvgw" event={"ID":"09a9e361-fc9a-4b67-93ab-49cad41d1d72","Type":"ContainerDied","Data":"4de97041f5222af3dda5327251b84e291e9db69a8195aa2525af6f241da71762"}
Jul 29 16:58:30.016149 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:30.015737    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4de97041f5222af3dda5327251b84e291e9db69a8195aa2525af6f241da71762"
Jul 29 16:58:35.249666 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249578    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-94qdj"]
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249895    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f759f258-3c4f-4d0d-8a56-f044033acf22" containerName="util"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249906    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f759f258-3c4f-4d0d-8a56-f044033acf22" containerName="util"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249915    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a9e361-fc9a-4b67-93ab-49cad41d1d72" containerName="extract"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249921    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a9e361-fc9a-4b67-93ab-49cad41d1d72" containerName="extract"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249933    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795ce7ec-2f54-4995-95c1-d41575a19860" containerName="pull"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249939    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="795ce7ec-2f54-4995-95c1-d41575a19860" containerName="pull"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249948    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5" containerName="util"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249953    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5" containerName="util"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249960    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5" containerName="extract"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249965    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5" containerName="extract"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249972    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795ce7ec-2f54-4995-95c1-d41575a19860" containerName="util"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249977    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="795ce7ec-2f54-4995-95c1-d41575a19860" containerName="util"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249984    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a9e361-fc9a-4b67-93ab-49cad41d1d72" containerName="pull"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249989    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a9e361-fc9a-4b67-93ab-49cad41d1d72" containerName="pull"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.249997    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5" containerName="pull"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.250002    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5" containerName="pull"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.250007    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f759f258-3c4f-4d0d-8a56-f044033acf22" containerName="pull"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.250012    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f759f258-3c4f-4d0d-8a56-f044033acf22" containerName="pull"
Jul 29 16:58:35.250018 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.250018    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f759f258-3c4f-4d0d-8a56-f044033acf22" containerName="extract"
Jul 29 16:58:35.250695 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.250023    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f759f258-3c4f-4d0d-8a56-f044033acf22" containerName="extract"
Jul 29 16:58:35.250695 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.250048    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795ce7ec-2f54-4995-95c1-d41575a19860" containerName="extract"
Jul 29 16:58:35.250695 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.250053    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="795ce7ec-2f54-4995-95c1-d41575a19860" containerName="extract"
Jul 29 16:58:35.250695 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.250061    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a9e361-fc9a-4b67-93ab-49cad41d1d72" containerName="util"
Jul 29 16:58:35.250695 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.250066    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a9e361-fc9a-4b67-93ab-49cad41d1d72" containerName="util"
Jul 29 16:58:35.250695 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.250110    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="09a9e361-fc9a-4b67-93ab-49cad41d1d72" containerName="extract"
Jul 29 16:58:35.250695 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.250119    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f759f258-3c4f-4d0d-8a56-f044033acf22" containerName="extract"
Jul 29 16:58:35.250695 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.250126    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9fb22be-f19b-4d9c-ade9-e7e8e22b06c5" containerName="extract"
Jul 29 16:58:35.250695 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.250132    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="795ce7ec-2f54-4995-95c1-d41575a19860" containerName="extract"
Jul 29 16:58:35.256542 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.254819    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-94qdj"
Jul 29 16:58:35.257904 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.257883    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 16:58:35.257970 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.257896    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 16:58:35.258039 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.258020    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-h2h4v\""
Jul 29 16:58:35.265509 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.265481    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-94qdj"]
Jul 29 16:58:35.407600 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.407562    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85jp8\" (UniqueName: \"kubernetes.io/projected/f62182af-14ff-4795-b4fd-34cf1ae2e435-kube-api-access-85jp8\") pod \"authorino-operator-56fb6f6856-94qdj\" (UID: \"f62182af-14ff-4795-b4fd-34cf1ae2e435\") " pod="kuadrant-system/authorino-operator-56fb6f6856-94qdj"
Jul 29 16:58:35.508827 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.508789    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85jp8\" (UniqueName: \"kubernetes.io/projected/f62182af-14ff-4795-b4fd-34cf1ae2e435-kube-api-access-85jp8\") pod \"authorino-operator-56fb6f6856-94qdj\" (UID: \"f62182af-14ff-4795-b4fd-34cf1ae2e435\") " pod="kuadrant-system/authorino-operator-56fb6f6856-94qdj"
Jul 29 16:58:35.517510 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.517478    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85jp8\" (UniqueName: \"kubernetes.io/projected/f62182af-14ff-4795-b4fd-34cf1ae2e435-kube-api-access-85jp8\") pod \"authorino-operator-56fb6f6856-94qdj\" (UID: \"f62182af-14ff-4795-b4fd-34cf1ae2e435\") " pod="kuadrant-system/authorino-operator-56fb6f6856-94qdj"
Jul 29 16:58:35.566502 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.566459    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-94qdj"
Jul 29 16:58:35.713472 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:35.713449    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-94qdj"]
Jul 29 16:58:35.715751 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:58:35.715714    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62182af_14ff_4795_b4fd_34cf1ae2e435.slice/crio-7b16b89ca015f36921312adbdc5135f237c16612ef6472511dde82ec955d97cb WatchSource:0}: Error finding container 7b16b89ca015f36921312adbdc5135f237c16612ef6472511dde82ec955d97cb: Status 404 returned error can't find the container with id 7b16b89ca015f36921312adbdc5135f237c16612ef6472511dde82ec955d97cb
Jul 29 16:58:36.046852 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:36.046813    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-94qdj" event={"ID":"f62182af-14ff-4795-b4fd-34cf1ae2e435","Type":"ContainerStarted","Data":"7b16b89ca015f36921312adbdc5135f237c16612ef6472511dde82ec955d97cb"}
Jul 29 16:58:39.068183 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:39.068128    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-94qdj" event={"ID":"f62182af-14ff-4795-b4fd-34cf1ae2e435","Type":"ContainerStarted","Data":"ffd3a6b8845a4a558d7c26fbbd470bcea69c017de6d6ec50f77bc95e4e30aaab"}
Jul 29 16:58:39.068594 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:39.068213    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-94qdj"
Jul 29 16:58:39.092808 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:39.092751    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-56fb6f6856-94qdj" podStartSLOduration=1.262237267 podStartE2EDuration="4.092729066s" totalImagesPullingTime="2.830491799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:58:39.091911189 +0000 UTC m=+447.581430846" watchObservedRunningTime="2026-07-29 16:58:39.092729066 +0000 UTC m=+447.582248725"
Jul 29 16:58:39.974935 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:39.974898    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-pc9v2"]
Jul 29 16:58:39.978516 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:39.978497    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-pc9v2"
Jul 29 16:58:39.981013 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:39.980985    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6z5pv\""
Jul 29 16:58:39.999604 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:39.999575    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-pc9v2"]
Jul 29 16:58:40.150179 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:40.150130    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc4ks\" (UniqueName: \"kubernetes.io/projected/0cc85d6c-1484-4066-b303-a2d2ae732521-kube-api-access-sc4ks\") pod \"limitador-operator-controller-manager-774d94476f-pc9v2\" (UID: \"0cc85d6c-1484-4066-b303-a2d2ae732521\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-pc9v2"
Jul 29 16:58:40.251012 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:40.250981    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc4ks\" (UniqueName: \"kubernetes.io/projected/0cc85d6c-1484-4066-b303-a2d2ae732521-kube-api-access-sc4ks\") pod \"limitador-operator-controller-manager-774d94476f-pc9v2\" (UID: \"0cc85d6c-1484-4066-b303-a2d2ae732521\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-pc9v2"
Jul 29 16:58:40.262118 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:40.262085    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc4ks\" (UniqueName: \"kubernetes.io/projected/0cc85d6c-1484-4066-b303-a2d2ae732521-kube-api-access-sc4ks\") pod \"limitador-operator-controller-manager-774d94476f-pc9v2\" (UID: \"0cc85d6c-1484-4066-b303-a2d2ae732521\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-pc9v2"
Jul 29 16:58:40.289206 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:40.289141    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-pc9v2"
Jul 29 16:58:40.463871 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:40.463847    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-pc9v2"]
Jul 29 16:58:40.465848 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:58:40.465818    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc85d6c_1484_4066_b303_a2d2ae732521.slice/crio-5725c89163706f2e384f1a32126c236a393dabbc84c64851e10fb054973df38c WatchSource:0}: Error finding container 5725c89163706f2e384f1a32126c236a393dabbc84c64851e10fb054973df38c: Status 404 returned error can't find the container with id 5725c89163706f2e384f1a32126c236a393dabbc84c64851e10fb054973df38c
Jul 29 16:58:41.080405 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:41.080363    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-pc9v2" event={"ID":"0cc85d6c-1484-4066-b303-a2d2ae732521","Type":"ContainerStarted","Data":"5725c89163706f2e384f1a32126c236a393dabbc84c64851e10fb054973df38c"}
Jul 29 16:58:42.087827 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:42.087797    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-pc9v2" event={"ID":"0cc85d6c-1484-4066-b303-a2d2ae732521","Type":"ContainerStarted","Data":"bac4840ad61fc13e4c2934c76b655bcf836e5812d21bb711b02c8eb34c3f8222"}
Jul 29 16:58:42.088233 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:42.087860    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-pc9v2"
Jul 29 16:58:42.108437 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:42.108380    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-pc9v2" podStartSLOduration=1.558251031 podStartE2EDuration="3.108363803s" totalImagesPullingTime="1.550112772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:58:42.106669189 +0000 UTC m=+450.596188904" watchObservedRunningTime="2026-07-29 16:58:42.108363803 +0000 UTC m=+450.597883459"
Jul 29 16:58:45.150767 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.150727    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5859d65dc9-4hlnh" podUID="aa6a2eeb-e489-4940-a5c2-f845e807b4b1" containerName="console" containerID="cri-o://57c500ed14184aad426cf5ea29298e1ac0f21810e7bf78f3d54fbd12bced1585" gracePeriod=15
Jul 29 16:58:45.416386 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.416356    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:58:45.597644 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.597608    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlktj\" (UniqueName: \"kubernetes.io/projected/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-kube-api-access-mlktj\") pod \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") "
Jul 29 16:58:45.597805 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.597660    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-serving-cert\") pod \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") "
Jul 29 16:58:45.597805 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.597707    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-service-ca\") pod \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") "
Jul 29 16:58:45.597805 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.597751    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-oauth-serving-cert\") pod \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") "
Jul 29 16:58:45.597805 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.597801    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-config\") pod \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") "
Jul 29 16:58:45.598005 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.597846    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-trusted-ca-bundle\") pod \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") "
Jul 29 16:58:45.598005 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.597890    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-oauth-config\") pod \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\" (UID: \"aa6a2eeb-e489-4940-a5c2-f845e807b4b1\") "
Jul 29 16:58:45.598333 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.598141    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-service-ca" (OuterVolumeSpecName: "service-ca") pod "aa6a2eeb-e489-4940-a5c2-f845e807b4b1" (UID: "aa6a2eeb-e489-4940-a5c2-f845e807b4b1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 16:58:45.598333 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.598252    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-config" (OuterVolumeSpecName: "console-config") pod "aa6a2eeb-e489-4940-a5c2-f845e807b4b1" (UID: "aa6a2eeb-e489-4940-a5c2-f845e807b4b1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 16:58:45.598475 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.598278    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aa6a2eeb-e489-4940-a5c2-f845e807b4b1" (UID: "aa6a2eeb-e489-4940-a5c2-f845e807b4b1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 16:58:45.598475 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.598346    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aa6a2eeb-e489-4940-a5c2-f845e807b4b1" (UID: "aa6a2eeb-e489-4940-a5c2-f845e807b4b1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 16:58:45.599927 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.599903    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-kube-api-access-mlktj" (OuterVolumeSpecName: "kube-api-access-mlktj") pod "aa6a2eeb-e489-4940-a5c2-f845e807b4b1" (UID: "aa6a2eeb-e489-4940-a5c2-f845e807b4b1"). InnerVolumeSpecName "kube-api-access-mlktj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:58:45.600076 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.600051    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aa6a2eeb-e489-4940-a5c2-f845e807b4b1" (UID: "aa6a2eeb-e489-4940-a5c2-f845e807b4b1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 16:58:45.600218 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.600094    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aa6a2eeb-e489-4940-a5c2-f845e807b4b1" (UID: "aa6a2eeb-e489-4940-a5c2-f845e807b4b1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 16:58:45.699079 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.698986    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlktj\" (UniqueName: \"kubernetes.io/projected/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-kube-api-access-mlktj\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:45.699079 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.699019    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-serving-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:45.699079 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.699028    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-service-ca\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:45.699079 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.699039    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-oauth-serving-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:45.699079 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.699048    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:45.699079 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.699056    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-trusted-ca-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:45.699079 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:45.699064    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa6a2eeb-e489-4940-a5c2-f845e807b4b1-console-oauth-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 16:58:46.108642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:46.108605    2570 generic.go:356] "Generic (PLEG): container finished" podID="aa6a2eeb-e489-4940-a5c2-f845e807b4b1" containerID="57c500ed14184aad426cf5ea29298e1ac0f21810e7bf78f3d54fbd12bced1585" exitCode=0
Jul 29 16:58:46.108642 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:46.108648    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5859d65dc9-4hlnh"
Jul 29 16:58:46.108922 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:46.108690    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5859d65dc9-4hlnh" event={"ID":"aa6a2eeb-e489-4940-a5c2-f845e807b4b1","Type":"ContainerDied","Data":"57c500ed14184aad426cf5ea29298e1ac0f21810e7bf78f3d54fbd12bced1585"}
Jul 29 16:58:46.108922 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:46.108756    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5859d65dc9-4hlnh" event={"ID":"aa6a2eeb-e489-4940-a5c2-f845e807b4b1","Type":"ContainerDied","Data":"c3cf104614226d66fa31a9a75ee0ee769d268ae6f49305dd38f232ee02f52c5b"}
Jul 29 16:58:46.108922 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:46.108785    2570 scope.go:117] "RemoveContainer" containerID="57c500ed14184aad426cf5ea29298e1ac0f21810e7bf78f3d54fbd12bced1585"
Jul 29 16:58:46.117703 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:46.117678    2570 scope.go:117] "RemoveContainer" containerID="57c500ed14184aad426cf5ea29298e1ac0f21810e7bf78f3d54fbd12bced1585"
Jul 29 16:58:46.118037 ip-10-0-134-156 kubenswrapper[2570]: E0729 16:58:46.118014    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c500ed14184aad426cf5ea29298e1ac0f21810e7bf78f3d54fbd12bced1585\": container with ID starting with 57c500ed14184aad426cf5ea29298e1ac0f21810e7bf78f3d54fbd12bced1585 not found: ID does not exist" containerID="57c500ed14184aad426cf5ea29298e1ac0f21810e7bf78f3d54fbd12bced1585"
Jul 29 16:58:46.118108 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:46.118051    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c500ed14184aad426cf5ea29298e1ac0f21810e7bf78f3d54fbd12bced1585"} err="failed to get container status \"57c500ed14184aad426cf5ea29298e1ac0f21810e7bf78f3d54fbd12bced1585\": rpc error: code = NotFound desc = could not find container \"57c500ed14184aad426cf5ea29298e1ac0f21810e7bf78f3d54fbd12bced1585\": container with ID starting with 57c500ed14184aad426cf5ea29298e1ac0f21810e7bf78f3d54fbd12bced1585 not found: ID does not exist"
Jul 29 16:58:46.129720 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:46.129686    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5859d65dc9-4hlnh"]
Jul 29 16:58:46.137285 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:46.137244    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5859d65dc9-4hlnh"]
Jul 29 16:58:47.913237 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:47.913199    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa6a2eeb-e489-4940-a5c2-f845e807b4b1" path="/var/lib/kubelet/pods/aa6a2eeb-e489-4940-a5c2-f845e807b4b1/volumes"
Jul 29 16:58:50.076660 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:50.076633    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-94qdj"
Jul 29 16:58:53.096177 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:58:53.096132    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-pc9v2"
Jul 29 16:59:36.364430 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.364390    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-q7znc"]
Jul 29 16:59:36.365063 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.364924    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa6a2eeb-e489-4940-a5c2-f845e807b4b1" containerName="console"
Jul 29 16:59:36.365063 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.364944    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa6a2eeb-e489-4940-a5c2-f845e807b4b1" containerName="console"
Jul 29 16:59:36.365063 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.365032    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa6a2eeb-e489-4940-a5c2-f845e807b4b1" containerName="console"
Jul 29 16:59:36.368060 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.368033    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-q7znc"
Jul 29 16:59:36.370776 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.370744    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 29 16:59:36.370936 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.370842    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wmsnf\""
Jul 29 16:59:36.377596 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.377565    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-q7znc"]
Jul 29 16:59:36.414679 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.414634    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-q7znc"]
Jul 29 16:59:36.533518 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.533478    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnxm4\" (UniqueName: \"kubernetes.io/projected/8d565869-182a-4c65-97a7-94053de639ef-kube-api-access-mnxm4\") pod \"limitador-limitador-85d9db4d5d-q7znc\" (UID: \"8d565869-182a-4c65-97a7-94053de639ef\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-q7znc"
Jul 29 16:59:36.533705 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.533532    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d565869-182a-4c65-97a7-94053de639ef-config-file\") pod \"limitador-limitador-85d9db4d5d-q7znc\" (UID: \"8d565869-182a-4c65-97a7-94053de639ef\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-q7znc"
Jul 29 16:59:36.634554 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.634463    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnxm4\" (UniqueName: \"kubernetes.io/projected/8d565869-182a-4c65-97a7-94053de639ef-kube-api-access-mnxm4\") pod \"limitador-limitador-85d9db4d5d-q7znc\" (UID: \"8d565869-182a-4c65-97a7-94053de639ef\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-q7znc"
Jul 29 16:59:36.634554 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.634510    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d565869-182a-4c65-97a7-94053de639ef-config-file\") pod \"limitador-limitador-85d9db4d5d-q7znc\" (UID: \"8d565869-182a-4c65-97a7-94053de639ef\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-q7znc"
Jul 29 16:59:36.635089 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.635069    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d565869-182a-4c65-97a7-94053de639ef-config-file\") pod \"limitador-limitador-85d9db4d5d-q7znc\" (UID: \"8d565869-182a-4c65-97a7-94053de639ef\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-q7znc"
Jul 29 16:59:36.642992 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.642968    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnxm4\" (UniqueName: \"kubernetes.io/projected/8d565869-182a-4c65-97a7-94053de639ef-kube-api-access-mnxm4\") pod \"limitador-limitador-85d9db4d5d-q7znc\" (UID: \"8d565869-182a-4c65-97a7-94053de639ef\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-q7znc"
Jul 29 16:59:36.678976 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.678941    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-q7znc"
Jul 29 16:59:36.829381 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:36.829344    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-q7znc"]
Jul 29 16:59:36.835633 ip-10-0-134-156 kubenswrapper[2570]: W0729 16:59:36.835461    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d565869_182a_4c65_97a7_94053de639ef.slice/crio-4aa1370e75cec1b49d57186991dd4a0e127e950310a0b2fa2965441794c098ef WatchSource:0}: Error finding container 4aa1370e75cec1b49d57186991dd4a0e127e950310a0b2fa2965441794c098ef: Status 404 returned error can't find the container with id 4aa1370e75cec1b49d57186991dd4a0e127e950310a0b2fa2965441794c098ef
Jul 29 16:59:37.379931 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:37.379894    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-q7znc" event={"ID":"8d565869-182a-4c65-97a7-94053de639ef","Type":"ContainerStarted","Data":"4aa1370e75cec1b49d57186991dd4a0e127e950310a0b2fa2965441794c098ef"}
Jul 29 16:59:38.387466 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:38.387424    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-q7znc" event={"ID":"8d565869-182a-4c65-97a7-94053de639ef","Type":"ContainerStarted","Data":"9f31704c4e60796023b995d9ba45bb4c1a330ec3760f553f311d38b4237154ef"}
Jul 29 16:59:38.387895 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:38.387595    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-q7znc"
Jul 29 16:59:38.409439 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:38.409381    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-q7znc" podStartSLOduration=0.972546062 podStartE2EDuration="2.409362543s" totalImagesPullingTime="1.436816481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:59:38.407397248 +0000 UTC m=+506.896916905" watchObservedRunningTime="2026-07-29 16:59:38.409362543 +0000 UTC m=+506.898882199"
Jul 29 16:59:49.393611 ip-10-0-134-156 kubenswrapper[2570]: I0729 16:59:49.393577    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-q7znc"
Jul 29 17:00:15.701648 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:15.701610    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"]
Jul 29 17:00:15.702137 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:15.701828    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm" podUID="167a4d72-f5d1-4be5-895d-1f96bea939c4" containerName="discovery" containerID="cri-o://44e02346bc5b0b9f841c4dc7f08d0b76a8afb42c0adaa5d271f568ebe9cdab16" gracePeriod=30
Jul 29 17:00:15.966176 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:15.966092    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 17:00:16.071828 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.071789    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68m7b\" (UniqueName: \"kubernetes.io/projected/167a4d72-f5d1-4be5-895d-1f96bea939c4-kube-api-access-68m7b\") pod \"167a4d72-f5d1-4be5-895d-1f96bea939c4\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") "
Jul 29 17:00:16.072011 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.071847    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/167a4d72-f5d1-4be5-895d-1f96bea939c4-local-certs\") pod \"167a4d72-f5d1-4be5-895d-1f96bea939c4\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") "
Jul 29 17:00:16.072011 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.071899    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-token\") pod \"167a4d72-f5d1-4be5-895d-1f96bea939c4\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") "
Jul 29 17:00:16.072011 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.071938    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-kubeconfig\") pod \"167a4d72-f5d1-4be5-895d-1f96bea939c4\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") "
Jul 29 17:00:16.072110 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.072067    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-cacerts\") pod \"167a4d72-f5d1-4be5-895d-1f96bea939c4\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") "
Jul 29 17:00:16.072155 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.072108    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-csr-dns-cert\") pod \"167a4d72-f5d1-4be5-895d-1f96bea939c4\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") "
Jul 29 17:00:16.072155 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.072141    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-csr-ca-configmap\") pod \"167a4d72-f5d1-4be5-895d-1f96bea939c4\" (UID: \"167a4d72-f5d1-4be5-895d-1f96bea939c4\") "
Jul 29 17:00:16.072751 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.072701    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "167a4d72-f5d1-4be5-895d-1f96bea939c4" (UID: "167a4d72-f5d1-4be5-895d-1f96bea939c4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 17:00:16.074467 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.074431    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "167a4d72-f5d1-4be5-895d-1f96bea939c4" (UID: "167a4d72-f5d1-4be5-895d-1f96bea939c4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:00:16.074586 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.074542    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167a4d72-f5d1-4be5-895d-1f96bea939c4-kube-api-access-68m7b" (OuterVolumeSpecName: "kube-api-access-68m7b") pod "167a4d72-f5d1-4be5-895d-1f96bea939c4" (UID: "167a4d72-f5d1-4be5-895d-1f96bea939c4"). InnerVolumeSpecName "kube-api-access-68m7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:00:16.074646 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.074629    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/167a4d72-f5d1-4be5-895d-1f96bea939c4-local-certs" (OuterVolumeSpecName: "local-certs") pod "167a4d72-f5d1-4be5-895d-1f96bea939c4" (UID: "167a4d72-f5d1-4be5-895d-1f96bea939c4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:00:16.074706 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.074682    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "167a4d72-f5d1-4be5-895d-1f96bea939c4" (UID: "167a4d72-f5d1-4be5-895d-1f96bea939c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:00:16.074755 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.074731    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-cacerts" (OuterVolumeSpecName: "cacerts") pod "167a4d72-f5d1-4be5-895d-1f96bea939c4" (UID: "167a4d72-f5d1-4be5-895d-1f96bea939c4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:00:16.074837 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.074822    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "167a4d72-f5d1-4be5-895d-1f96bea939c4" (UID: "167a4d72-f5d1-4be5-895d-1f96bea939c4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:00:16.173858 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.173692    2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-csr-dns-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:00:16.173858 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.173734    2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-csr-ca-configmap\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:00:16.173858 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.173748    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68m7b\" (UniqueName: \"kubernetes.io/projected/167a4d72-f5d1-4be5-895d-1f96bea939c4-kube-api-access-68m7b\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:00:16.173858 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.173761    2570 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/167a4d72-f5d1-4be5-895d-1f96bea939c4-local-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:00:16.173858 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.173777    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-token\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:00:16.173858 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.173789    2570 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-istio-kubeconfig\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:00:16.173858 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.173801    2570 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/167a4d72-f5d1-4be5-895d-1f96bea939c4-cacerts\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:00:16.582773 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.582731    2570 generic.go:356] "Generic (PLEG): container finished" podID="167a4d72-f5d1-4be5-895d-1f96bea939c4" containerID="44e02346bc5b0b9f841c4dc7f08d0b76a8afb42c0adaa5d271f568ebe9cdab16" exitCode=0
Jul 29 17:00:16.582773 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.582777    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"
Jul 29 17:00:16.582993 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.582797    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm" event={"ID":"167a4d72-f5d1-4be5-895d-1f96bea939c4","Type":"ContainerDied","Data":"44e02346bc5b0b9f841c4dc7f08d0b76a8afb42c0adaa5d271f568ebe9cdab16"}
Jul 29 17:00:16.582993 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.582846    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm" event={"ID":"167a4d72-f5d1-4be5-895d-1f96bea939c4","Type":"ContainerDied","Data":"48ef05df30c269179a44ccd42c3c44f8c22c944f37f459b820439cbc23b2c367"}
Jul 29 17:00:16.582993 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.582869    2570 scope.go:117] "RemoveContainer" containerID="44e02346bc5b0b9f841c4dc7f08d0b76a8afb42c0adaa5d271f568ebe9cdab16"
Jul 29 17:00:16.592150 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.592123    2570 scope.go:117] "RemoveContainer" containerID="44e02346bc5b0b9f841c4dc7f08d0b76a8afb42c0adaa5d271f568ebe9cdab16"
Jul 29 17:00:16.592513 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:00:16.592497    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e02346bc5b0b9f841c4dc7f08d0b76a8afb42c0adaa5d271f568ebe9cdab16\": container with ID starting with 44e02346bc5b0b9f841c4dc7f08d0b76a8afb42c0adaa5d271f568ebe9cdab16 not found: ID does not exist" containerID="44e02346bc5b0b9f841c4dc7f08d0b76a8afb42c0adaa5d271f568ebe9cdab16"
Jul 29 17:00:16.592572 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.592523    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e02346bc5b0b9f841c4dc7f08d0b76a8afb42c0adaa5d271f568ebe9cdab16"} err="failed to get container status \"44e02346bc5b0b9f841c4dc7f08d0b76a8afb42c0adaa5d271f568ebe9cdab16\": rpc error: code = NotFound desc = could not find container \"44e02346bc5b0b9f841c4dc7f08d0b76a8afb42c0adaa5d271f568ebe9cdab16\": container with ID starting with 44e02346bc5b0b9f841c4dc7f08d0b76a8afb42c0adaa5d271f568ebe9cdab16 not found: ID does not exist"
Jul 29 17:00:16.605845 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.605809    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"]
Jul 29 17:00:16.611079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:16.611045    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hxqvm"]
Jul 29 17:00:17.913484 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:17.913451    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="167a4d72-f5d1-4be5-895d-1f96bea939c4" path="/var/lib/kubelet/pods/167a4d72-f5d1-4be5-895d-1f96bea939c4/volumes"
Jul 29 17:00:19.642843 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.642795    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cd5d4f59d-tf5k4"]
Jul 29 17:00:19.643400 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.643326    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="167a4d72-f5d1-4be5-895d-1f96bea939c4" containerName="discovery"
Jul 29 17:00:19.643400 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.643345    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="167a4d72-f5d1-4be5-895d-1f96bea939c4" containerName="discovery"
Jul 29 17:00:19.643525 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.643455    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="167a4d72-f5d1-4be5-895d-1f96bea939c4" containerName="discovery"
Jul 29 17:00:19.648464 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.648437    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4"
Jul 29 17:00:19.651026 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.650993    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 29 17:00:19.651355 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.651335    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-qfg2q\""
Jul 29 17:00:19.651484 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.651339    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 29 17:00:19.652315 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.652285    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 29 17:00:19.659138 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.659107    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd5d4f59d-tf5k4"]
Jul 29 17:00:19.664185 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.664132    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84b6684489-mlktn"]
Jul 29 17:00:19.668052 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.668025    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84b6684489-mlktn"
Jul 29 17:00:19.671352 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.671119    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 29 17:00:19.671352 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.671325    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-q25wn\""
Jul 29 17:00:19.678233 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.678207    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84b6684489-mlktn"]
Jul 29 17:00:19.695634 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.695602    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-8445dbf85d-s6kvn"]
Jul 29 17:00:19.699402 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.699367    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-8445dbf85d-s6kvn"
Jul 29 17:00:19.702285 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.702262    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-qtkwt\""
Jul 29 17:00:19.702412 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.702283    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 29 17:00:19.711962 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.711929    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-8445dbf85d-s6kvn"]
Jul 29 17:00:19.804110 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.804063    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6sl8\" (UniqueName: \"kubernetes.io/projected/dcb5831c-8598-4e15-9d60-264f462dbb67-kube-api-access-p6sl8\") pod \"kserve-controller-manager-cd5d4f59d-tf5k4\" (UID: \"dcb5831c-8598-4e15-9d60-264f462dbb67\") " pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4"
Jul 29 17:00:19.804305 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.804131    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5faf75f6-2825-4b36-9e84-0e52a96b2085-cert\") pod \"llmisvc-controller-manager-84b6684489-mlktn\" (UID: \"5faf75f6-2825-4b36-9e84-0e52a96b2085\") " pod="kserve/llmisvc-controller-manager-84b6684489-mlktn"
Jul 29 17:00:19.804305 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.804207    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcb5831c-8598-4e15-9d60-264f462dbb67-cert\") pod \"kserve-controller-manager-cd5d4f59d-tf5k4\" (UID: \"dcb5831c-8598-4e15-9d60-264f462dbb67\") " pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4"
Jul 29 17:00:19.804305 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.804231    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/db91b8cd-9b2d-4caa-867b-c32a4346a28b-data\") pod \"seaweedfs-8445dbf85d-s6kvn\" (UID: \"db91b8cd-9b2d-4caa-867b-c32a4346a28b\") " pod="kserve/seaweedfs-8445dbf85d-s6kvn"
Jul 29 17:00:19.804305 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.804252    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k5jg\" (UniqueName: \"kubernetes.io/projected/db91b8cd-9b2d-4caa-867b-c32a4346a28b-kube-api-access-5k5jg\") pod \"seaweedfs-8445dbf85d-s6kvn\" (UID: \"db91b8cd-9b2d-4caa-867b-c32a4346a28b\") " pod="kserve/seaweedfs-8445dbf85d-s6kvn"
Jul 29 17:00:19.804305 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.804268    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb9sr\" (UniqueName: \"kubernetes.io/projected/5faf75f6-2825-4b36-9e84-0e52a96b2085-kube-api-access-mb9sr\") pod \"llmisvc-controller-manager-84b6684489-mlktn\" (UID: \"5faf75f6-2825-4b36-9e84-0e52a96b2085\") " pod="kserve/llmisvc-controller-manager-84b6684489-mlktn"
Jul 29 17:00:19.905292 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.905156    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5faf75f6-2825-4b36-9e84-0e52a96b2085-cert\") pod \"llmisvc-controller-manager-84b6684489-mlktn\" (UID: \"5faf75f6-2825-4b36-9e84-0e52a96b2085\") " pod="kserve/llmisvc-controller-manager-84b6684489-mlktn"
Jul 29 17:00:19.905292 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.905259    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcb5831c-8598-4e15-9d60-264f462dbb67-cert\") pod \"kserve-controller-manager-cd5d4f59d-tf5k4\" (UID: \"dcb5831c-8598-4e15-9d60-264f462dbb67\") " pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4"
Jul 29 17:00:19.905292 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.905287    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/db91b8cd-9b2d-4caa-867b-c32a4346a28b-data\") pod \"seaweedfs-8445dbf85d-s6kvn\" (UID: \"db91b8cd-9b2d-4caa-867b-c32a4346a28b\") " pod="kserve/seaweedfs-8445dbf85d-s6kvn"
Jul 29 17:00:19.905576 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:00:19.905306    2570 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 29 17:00:19.905576 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:00:19.905392    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5faf75f6-2825-4b36-9e84-0e52a96b2085-cert podName:5faf75f6-2825-4b36-9e84-0e52a96b2085 nodeName:}" failed. No retries permitted until 2026-07-29 17:00:20.40537483 +0000 UTC m=+548.894894467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5faf75f6-2825-4b36-9e84-0e52a96b2085-cert") pod "llmisvc-controller-manager-84b6684489-mlktn" (UID: "5faf75f6-2825-4b36-9e84-0e52a96b2085") : secret "llmisvc-webhook-server-cert" not found
Jul 29 17:00:19.905576 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.905310    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k5jg\" (UniqueName: \"kubernetes.io/projected/db91b8cd-9b2d-4caa-867b-c32a4346a28b-kube-api-access-5k5jg\") pod \"seaweedfs-8445dbf85d-s6kvn\" (UID: \"db91b8cd-9b2d-4caa-867b-c32a4346a28b\") " pod="kserve/seaweedfs-8445dbf85d-s6kvn"
Jul 29 17:00:19.905576 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.905443    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb9sr\" (UniqueName: \"kubernetes.io/projected/5faf75f6-2825-4b36-9e84-0e52a96b2085-kube-api-access-mb9sr\") pod \"llmisvc-controller-manager-84b6684489-mlktn\" (UID: \"5faf75f6-2825-4b36-9e84-0e52a96b2085\") " pod="kserve/llmisvc-controller-manager-84b6684489-mlktn"
Jul 29 17:00:19.905576 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:00:19.905392    2570 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 29 17:00:19.905576 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.905488    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6sl8\" (UniqueName: \"kubernetes.io/projected/dcb5831c-8598-4e15-9d60-264f462dbb67-kube-api-access-p6sl8\") pod \"kserve-controller-manager-cd5d4f59d-tf5k4\" (UID: \"dcb5831c-8598-4e15-9d60-264f462dbb67\") " pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4"
Jul 29 17:00:19.905858 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:00:19.905585    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcb5831c-8598-4e15-9d60-264f462dbb67-cert podName:dcb5831c-8598-4e15-9d60-264f462dbb67 nodeName:}" failed. No retries permitted until 2026-07-29 17:00:20.405568781 +0000 UTC m=+548.895088430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dcb5831c-8598-4e15-9d60-264f462dbb67-cert") pod "kserve-controller-manager-cd5d4f59d-tf5k4" (UID: "dcb5831c-8598-4e15-9d60-264f462dbb67") : secret "kserve-webhook-server-cert" not found
Jul 29 17:00:19.905858 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.905677    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/db91b8cd-9b2d-4caa-867b-c32a4346a28b-data\") pod \"seaweedfs-8445dbf85d-s6kvn\" (UID: \"db91b8cd-9b2d-4caa-867b-c32a4346a28b\") " pod="kserve/seaweedfs-8445dbf85d-s6kvn"
Jul 29 17:00:19.917178 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.917134    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb9sr\" (UniqueName: \"kubernetes.io/projected/5faf75f6-2825-4b36-9e84-0e52a96b2085-kube-api-access-mb9sr\") pod \"llmisvc-controller-manager-84b6684489-mlktn\" (UID: \"5faf75f6-2825-4b36-9e84-0e52a96b2085\") " pod="kserve/llmisvc-controller-manager-84b6684489-mlktn"
Jul 29 17:00:19.917334 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.917266    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k5jg\" (UniqueName: \"kubernetes.io/projected/db91b8cd-9b2d-4caa-867b-c32a4346a28b-kube-api-access-5k5jg\") pod \"seaweedfs-8445dbf85d-s6kvn\" (UID: \"db91b8cd-9b2d-4caa-867b-c32a4346a28b\") " pod="kserve/seaweedfs-8445dbf85d-s6kvn"
Jul 29 17:00:19.917334 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:19.917295    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6sl8\" (UniqueName: \"kubernetes.io/projected/dcb5831c-8598-4e15-9d60-264f462dbb67-kube-api-access-p6sl8\") pod \"kserve-controller-manager-cd5d4f59d-tf5k4\" (UID: \"dcb5831c-8598-4e15-9d60-264f462dbb67\") " pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4"
Jul 29 17:00:20.009233 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:20.009193    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-8445dbf85d-s6kvn"
Jul 29 17:00:20.151233 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:20.151205    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-8445dbf85d-s6kvn"]
Jul 29 17:00:20.153009 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:00:20.152969    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb91b8cd_9b2d_4caa_867b_c32a4346a28b.slice/crio-abab7e5e53db7d4649869614606795c6088db8fb276c13228dee2472f77e21b9 WatchSource:0}: Error finding container abab7e5e53db7d4649869614606795c6088db8fb276c13228dee2472f77e21b9: Status 404 returned error can't find the container with id abab7e5e53db7d4649869614606795c6088db8fb276c13228dee2472f77e21b9
Jul 29 17:00:20.410707 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:20.410612    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcb5831c-8598-4e15-9d60-264f462dbb67-cert\") pod \"kserve-controller-manager-cd5d4f59d-tf5k4\" (UID: \"dcb5831c-8598-4e15-9d60-264f462dbb67\") " pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4"
Jul 29 17:00:20.410707 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:20.410686    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5faf75f6-2825-4b36-9e84-0e52a96b2085-cert\") pod \"llmisvc-controller-manager-84b6684489-mlktn\" (UID: \"5faf75f6-2825-4b36-9e84-0e52a96b2085\") " pod="kserve/llmisvc-controller-manager-84b6684489-mlktn"
Jul 29 17:00:20.413241 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:20.413215    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcb5831c-8598-4e15-9d60-264f462dbb67-cert\") pod \"kserve-controller-manager-cd5d4f59d-tf5k4\" (UID: \"dcb5831c-8598-4e15-9d60-264f462dbb67\") " pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4"
Jul 29 17:00:20.413241 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:20.413226    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5faf75f6-2825-4b36-9e84-0e52a96b2085-cert\") pod \"llmisvc-controller-manager-84b6684489-mlktn\" (UID: \"5faf75f6-2825-4b36-9e84-0e52a96b2085\") " pod="kserve/llmisvc-controller-manager-84b6684489-mlktn"
Jul 29 17:00:20.560114 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:20.560077    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4"
Jul 29 17:00:20.580266 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:20.580201    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84b6684489-mlktn"
Jul 29 17:00:20.609035 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:20.608997    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-8445dbf85d-s6kvn" event={"ID":"db91b8cd-9b2d-4caa-867b-c32a4346a28b","Type":"ContainerStarted","Data":"abab7e5e53db7d4649869614606795c6088db8fb276c13228dee2472f77e21b9"}
Jul 29 17:00:20.741606 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:20.741565    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd5d4f59d-tf5k4"]
Jul 29 17:00:20.776111 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:20.776026    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84b6684489-mlktn"]
Jul 29 17:00:20.811691 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:00:20.811650    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5faf75f6_2825_4b36_9e84_0e52a96b2085.slice/crio-f0f19161974e131f92a27daeba882a0987dd4f1bc18a8d5094335b6ea2556354 WatchSource:0}: Error finding container f0f19161974e131f92a27daeba882a0987dd4f1bc18a8d5094335b6ea2556354: Status 404 returned error can't find the container with id f0f19161974e131f92a27daeba882a0987dd4f1bc18a8d5094335b6ea2556354
Jul 29 17:00:20.813012 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:00:20.812979    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb5831c_8598_4e15_9d60_264f462dbb67.slice/crio-a76c30b55085a59a94a6e6c47aa72fa60bcfcd601c93712e18ee37ebc45df01a WatchSource:0}: Error finding container a76c30b55085a59a94a6e6c47aa72fa60bcfcd601c93712e18ee37ebc45df01a: Status 404 returned error can't find the container with id a76c30b55085a59a94a6e6c47aa72fa60bcfcd601c93712e18ee37ebc45df01a
Jul 29 17:00:21.617831 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:21.617786    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84b6684489-mlktn" event={"ID":"5faf75f6-2825-4b36-9e84-0e52a96b2085","Type":"ContainerStarted","Data":"f0f19161974e131f92a27daeba882a0987dd4f1bc18a8d5094335b6ea2556354"}
Jul 29 17:00:21.619216 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:21.619178    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4" event={"ID":"dcb5831c-8598-4e15-9d60-264f462dbb67","Type":"ContainerStarted","Data":"a76c30b55085a59a94a6e6c47aa72fa60bcfcd601c93712e18ee37ebc45df01a"}
Jul 29 17:00:25.646357 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:25.646319    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84b6684489-mlktn" event={"ID":"5faf75f6-2825-4b36-9e84-0e52a96b2085","Type":"ContainerStarted","Data":"cef85d7a2befbd221322f446a977a8bd10a59985180a6f37e36a00305bb1cb29"}
Jul 29 17:00:25.646805 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:25.646425    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84b6684489-mlktn"
Jul 29 17:00:25.648020 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:25.647994    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4" event={"ID":"dcb5831c-8598-4e15-9d60-264f462dbb67","Type":"ContainerStarted","Data":"dc7736fb330ce2926856563dfb211055d98e580bc1c68106f4a1f252f82d8ad6"}
Jul 29 17:00:25.648146 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:25.648048    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4"
Jul 29 17:00:25.649507 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:25.649485    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-8445dbf85d-s6kvn" event={"ID":"db91b8cd-9b2d-4caa-867b-c32a4346a28b","Type":"ContainerStarted","Data":"b4e6d85b5104db9dbd1e6d15130137b4278946687bdcea5cea64c95199c54a87"}
Jul 29 17:00:25.649616 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:25.649542    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-8445dbf85d-s6kvn"
Jul 29 17:00:25.665609 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:25.665558    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84b6684489-mlktn" podStartSLOduration=2.368931155 podStartE2EDuration="6.6655406s" totalImagesPullingTime="4.296609445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:00:25.66382559 +0000 UTC m=+554.153345247" watchObservedRunningTime="2026-07-29 17:00:25.6655406 +0000 UTC m=+554.155060279"
Jul 29 17:00:25.681444 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:25.681394    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-8445dbf85d-s6kvn" podStartSLOduration=1.581106268 podStartE2EDuration="6.681372568s" totalImagesPullingTime="5.1002663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:00:25.679751994 +0000 UTC m=+554.169271651" watchObservedRunningTime="2026-07-29 17:00:25.681372568 +0000 UTC m=+554.170892223"
Jul 29 17:00:25.698572 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:25.698512    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4" podStartSLOduration=2.358961731 podStartE2EDuration="6.698495875s" totalImagesPullingTime="4.339534144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:00:25.696363706 +0000 UTC m=+554.185883365" watchObservedRunningTime="2026-07-29 17:00:25.698495875 +0000 UTC m=+554.188015531"
Jul 29 17:00:31.657070 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:31.657039    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-8445dbf85d-s6kvn"
Jul 29 17:00:56.657311 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:56.657279    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84b6684489-mlktn"
Jul 29 17:00:56.660492 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:56.660474    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4"
Jul 29 17:00:58.175638 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.175601    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cd5d4f59d-tf5k4"]
Jul 29 17:00:58.176046 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.175790    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4" podUID="dcb5831c-8598-4e15-9d60-264f462dbb67" containerName="manager" containerID="cri-o://dc7736fb330ce2926856563dfb211055d98e580bc1c68106f4a1f252f82d8ad6" gracePeriod=10
Jul 29 17:00:58.204676 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.204644    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cd5d4f59d-4f2hx"]
Jul 29 17:00:58.207789 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.207770    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd5d4f59d-4f2hx"
Jul 29 17:00:58.222759 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.222726    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd5d4f59d-4f2hx"]
Jul 29 17:00:58.327769 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.327727    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26130c76-a292-4866-a4f1-2d254fc60d0b-cert\") pod \"kserve-controller-manager-cd5d4f59d-4f2hx\" (UID: \"26130c76-a292-4866-a4f1-2d254fc60d0b\") " pod="kserve/kserve-controller-manager-cd5d4f59d-4f2hx"
Jul 29 17:00:58.328013 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.327809    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4rj7\" (UniqueName: \"kubernetes.io/projected/26130c76-a292-4866-a4f1-2d254fc60d0b-kube-api-access-k4rj7\") pod \"kserve-controller-manager-cd5d4f59d-4f2hx\" (UID: \"26130c76-a292-4866-a4f1-2d254fc60d0b\") " pod="kserve/kserve-controller-manager-cd5d4f59d-4f2hx"
Jul 29 17:00:58.424841 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.424817    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4"
Jul 29 17:00:58.428964 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.428915    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26130c76-a292-4866-a4f1-2d254fc60d0b-cert\") pod \"kserve-controller-manager-cd5d4f59d-4f2hx\" (UID: \"26130c76-a292-4866-a4f1-2d254fc60d0b\") " pod="kserve/kserve-controller-manager-cd5d4f59d-4f2hx"
Jul 29 17:00:58.429045 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.428979    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4rj7\" (UniqueName: \"kubernetes.io/projected/26130c76-a292-4866-a4f1-2d254fc60d0b-kube-api-access-k4rj7\") pod \"kserve-controller-manager-cd5d4f59d-4f2hx\" (UID: \"26130c76-a292-4866-a4f1-2d254fc60d0b\") " pod="kserve/kserve-controller-manager-cd5d4f59d-4f2hx"
Jul 29 17:00:58.431401 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.431371    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26130c76-a292-4866-a4f1-2d254fc60d0b-cert\") pod \"kserve-controller-manager-cd5d4f59d-4f2hx\" (UID: \"26130c76-a292-4866-a4f1-2d254fc60d0b\") " pod="kserve/kserve-controller-manager-cd5d4f59d-4f2hx"
Jul 29 17:00:58.439904 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.439869    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4rj7\" (UniqueName: \"kubernetes.io/projected/26130c76-a292-4866-a4f1-2d254fc60d0b-kube-api-access-k4rj7\") pod \"kserve-controller-manager-cd5d4f59d-4f2hx\" (UID: \"26130c76-a292-4866-a4f1-2d254fc60d0b\") " pod="kserve/kserve-controller-manager-cd5d4f59d-4f2hx"
Jul 29 17:00:58.529839 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.529801    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcb5831c-8598-4e15-9d60-264f462dbb67-cert\") pod \"dcb5831c-8598-4e15-9d60-264f462dbb67\" (UID: \"dcb5831c-8598-4e15-9d60-264f462dbb67\") "
Jul 29 17:00:58.530016 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.529897    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6sl8\" (UniqueName: \"kubernetes.io/projected/dcb5831c-8598-4e15-9d60-264f462dbb67-kube-api-access-p6sl8\") pod \"dcb5831c-8598-4e15-9d60-264f462dbb67\" (UID: \"dcb5831c-8598-4e15-9d60-264f462dbb67\") "
Jul 29 17:00:58.532075 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.532046    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb5831c-8598-4e15-9d60-264f462dbb67-kube-api-access-p6sl8" (OuterVolumeSpecName: "kube-api-access-p6sl8") pod "dcb5831c-8598-4e15-9d60-264f462dbb67" (UID: "dcb5831c-8598-4e15-9d60-264f462dbb67"). InnerVolumeSpecName "kube-api-access-p6sl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:00:58.532075 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.532055    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb5831c-8598-4e15-9d60-264f462dbb67-cert" (OuterVolumeSpecName: "cert") pod "dcb5831c-8598-4e15-9d60-264f462dbb67" (UID: "dcb5831c-8598-4e15-9d60-264f462dbb67"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:00:58.558973 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.558938    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd5d4f59d-4f2hx"
Jul 29 17:00:58.631123 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.631090    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6sl8\" (UniqueName: \"kubernetes.io/projected/dcb5831c-8598-4e15-9d60-264f462dbb67-kube-api-access-p6sl8\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:00:58.631123 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.631120    2570 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcb5831c-8598-4e15-9d60-264f462dbb67-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:00:58.700065 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.700038    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd5d4f59d-4f2hx"]
Jul 29 17:00:58.700921 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:00:58.700896    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26130c76_a292_4866_a4f1_2d254fc60d0b.slice/crio-33fe26e78c60d26966feeee91469c7386351a5aeb956b52351d7d06d292db248 WatchSource:0}: Error finding container 33fe26e78c60d26966feeee91469c7386351a5aeb956b52351d7d06d292db248: Status 404 returned error can't find the container with id 33fe26e78c60d26966feeee91469c7386351a5aeb956b52351d7d06d292db248
Jul 29 17:00:58.820633 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.820597    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd5d4f59d-4f2hx" event={"ID":"26130c76-a292-4866-a4f1-2d254fc60d0b","Type":"ContainerStarted","Data":"33fe26e78c60d26966feeee91469c7386351a5aeb956b52351d7d06d292db248"}
Jul 29 17:00:58.821939 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.821916    2570 generic.go:356] "Generic (PLEG): container finished" podID="dcb5831c-8598-4e15-9d60-264f462dbb67" containerID="dc7736fb330ce2926856563dfb211055d98e580bc1c68106f4a1f252f82d8ad6" exitCode=0
Jul 29 17:00:58.822029 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.821953    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4" event={"ID":"dcb5831c-8598-4e15-9d60-264f462dbb67","Type":"ContainerDied","Data":"dc7736fb330ce2926856563dfb211055d98e580bc1c68106f4a1f252f82d8ad6"}
Jul 29 17:00:58.822029 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.821962    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4"
Jul 29 17:00:58.822029 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.821982    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd5d4f59d-tf5k4" event={"ID":"dcb5831c-8598-4e15-9d60-264f462dbb67","Type":"ContainerDied","Data":"a76c30b55085a59a94a6e6c47aa72fa60bcfcd601c93712e18ee37ebc45df01a"}
Jul 29 17:00:58.822029 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.822003    2570 scope.go:117] "RemoveContainer" containerID="dc7736fb330ce2926856563dfb211055d98e580bc1c68106f4a1f252f82d8ad6"
Jul 29 17:00:58.831413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.831393    2570 scope.go:117] "RemoveContainer" containerID="dc7736fb330ce2926856563dfb211055d98e580bc1c68106f4a1f252f82d8ad6"
Jul 29 17:00:58.831761 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:00:58.831729    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc7736fb330ce2926856563dfb211055d98e580bc1c68106f4a1f252f82d8ad6\": container with ID starting with dc7736fb330ce2926856563dfb211055d98e580bc1c68106f4a1f252f82d8ad6 not found: ID does not exist" containerID="dc7736fb330ce2926856563dfb211055d98e580bc1c68106f4a1f252f82d8ad6"
Jul 29 17:00:58.831815 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.831789    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc7736fb330ce2926856563dfb211055d98e580bc1c68106f4a1f252f82d8ad6"} err="failed to get container status \"dc7736fb330ce2926856563dfb211055d98e580bc1c68106f4a1f252f82d8ad6\": rpc error: code = NotFound desc = could not find container \"dc7736fb330ce2926856563dfb211055d98e580bc1c68106f4a1f252f82d8ad6\": container with ID starting with dc7736fb330ce2926856563dfb211055d98e580bc1c68106f4a1f252f82d8ad6 not found: ID does not exist"
Jul 29 17:00:58.846759 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.846713    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cd5d4f59d-tf5k4"]
Jul 29 17:00:58.850870 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:58.850842    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-cd5d4f59d-tf5k4"]
Jul 29 17:00:59.913953 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:00:59.913922    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcb5831c-8598-4e15-9d60-264f462dbb67" path="/var/lib/kubelet/pods/dcb5831c-8598-4e15-9d60-264f462dbb67/volumes"
Jul 29 17:01:00.839940 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:00.839902    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd5d4f59d-4f2hx" event={"ID":"26130c76-a292-4866-a4f1-2d254fc60d0b","Type":"ContainerStarted","Data":"d6e72ee880d8e4affbe8332aa89861c62191306ecd85ea1314e9477f53297c28"}
Jul 29 17:01:00.840104 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:00.839970    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cd5d4f59d-4f2hx"
Jul 29 17:01:00.859806 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:00.859754    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cd5d4f59d-4f2hx" podStartSLOduration=0.878401759 podStartE2EDuration="2.85973765s" totalImagesPullingTime="1.981335891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:01:00.857315345 +0000 UTC m=+589.346835003" watchObservedRunningTime="2026-07-29 17:01:00.85973765 +0000 UTC m=+589.349257316"
Jul 29 17:01:11.846885 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:11.846852    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:01:11.847369 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:11.846974    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:01:31.851491 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:31.851456    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cd5d4f59d-4f2hx"
Jul 29 17:01:32.889846 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:32.889808    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-k9vr4"]
Jul 29 17:01:32.890270 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:32.890182    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcb5831c-8598-4e15-9d60-264f462dbb67" containerName="manager"
Jul 29 17:01:32.890270 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:32.890194    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcb5831c-8598-4e15-9d60-264f462dbb67" containerName="manager"
Jul 29 17:01:32.890270 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:32.890256    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcb5831c-8598-4e15-9d60-264f462dbb67" containerName="manager"
Jul 29 17:01:32.893343 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:32.893317    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-k9vr4"
Jul 29 17:01:32.895559 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:32.895536    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 29 17:01:32.895559 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:32.895554    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-l2b2z\""
Jul 29 17:01:32.899952 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:32.899916    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-kl5hm"]
Jul 29 17:01:32.904647 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:32.904620    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-k9vr4"]
Jul 29 17:01:32.904847 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:32.904773    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-kl5hm"
Jul 29 17:01:32.907446 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:32.907420    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-hkrmv\""
Jul 29 17:01:32.907577 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:32.907420    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 29 17:01:32.916183 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:32.916140    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-kl5hm"]
Jul 29 17:01:33.035825 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.035784    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txqb7\" (UniqueName: \"kubernetes.io/projected/31aa38fa-7343-498c-8e17-9a7fd0c07b7b-kube-api-access-txqb7\") pod \"model-serving-api-7b954ffc95-k9vr4\" (UID: \"31aa38fa-7343-498c-8e17-9a7fd0c07b7b\") " pod="kserve/model-serving-api-7b954ffc95-k9vr4"
Jul 29 17:01:33.036012 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.035845    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk5zg\" (UniqueName: \"kubernetes.io/projected/3e3914fc-3283-494d-9546-4ef0cffb78aa-kube-api-access-kk5zg\") pod \"odh-model-controller-75d9cdbddf-kl5hm\" (UID: \"3e3914fc-3283-494d-9546-4ef0cffb78aa\") " pod="kserve/odh-model-controller-75d9cdbddf-kl5hm"
Jul 29 17:01:33.036012 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.035918    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e3914fc-3283-494d-9546-4ef0cffb78aa-cert\") pod \"odh-model-controller-75d9cdbddf-kl5hm\" (UID: \"3e3914fc-3283-494d-9546-4ef0cffb78aa\") " pod="kserve/odh-model-controller-75d9cdbddf-kl5hm"
Jul 29 17:01:33.036012 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.035964    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31aa38fa-7343-498c-8e17-9a7fd0c07b7b-tls-certs\") pod \"model-serving-api-7b954ffc95-k9vr4\" (UID: \"31aa38fa-7343-498c-8e17-9a7fd0c07b7b\") " pod="kserve/model-serving-api-7b954ffc95-k9vr4"
Jul 29 17:01:33.136434 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.136396    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txqb7\" (UniqueName: \"kubernetes.io/projected/31aa38fa-7343-498c-8e17-9a7fd0c07b7b-kube-api-access-txqb7\") pod \"model-serving-api-7b954ffc95-k9vr4\" (UID: \"31aa38fa-7343-498c-8e17-9a7fd0c07b7b\") " pod="kserve/model-serving-api-7b954ffc95-k9vr4"
Jul 29 17:01:33.136610 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.136441    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk5zg\" (UniqueName: \"kubernetes.io/projected/3e3914fc-3283-494d-9546-4ef0cffb78aa-kube-api-access-kk5zg\") pod \"odh-model-controller-75d9cdbddf-kl5hm\" (UID: \"3e3914fc-3283-494d-9546-4ef0cffb78aa\") " pod="kserve/odh-model-controller-75d9cdbddf-kl5hm"
Jul 29 17:01:33.136610 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.136475    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e3914fc-3283-494d-9546-4ef0cffb78aa-cert\") pod \"odh-model-controller-75d9cdbddf-kl5hm\" (UID: \"3e3914fc-3283-494d-9546-4ef0cffb78aa\") " pod="kserve/odh-model-controller-75d9cdbddf-kl5hm"
Jul 29 17:01:33.136610 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.136502    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31aa38fa-7343-498c-8e17-9a7fd0c07b7b-tls-certs\") pod \"model-serving-api-7b954ffc95-k9vr4\" (UID: \"31aa38fa-7343-498c-8e17-9a7fd0c07b7b\") " pod="kserve/model-serving-api-7b954ffc95-k9vr4"
Jul 29 17:01:33.136714 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:01:33.136687    2570 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 29 17:01:33.136780 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:01:33.136768    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31aa38fa-7343-498c-8e17-9a7fd0c07b7b-tls-certs podName:31aa38fa-7343-498c-8e17-9a7fd0c07b7b nodeName:}" failed. No retries permitted until 2026-07-29 17:01:33.636747291 +0000 UTC m=+622.126266931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/31aa38fa-7343-498c-8e17-9a7fd0c07b7b-tls-certs") pod "model-serving-api-7b954ffc95-k9vr4" (UID: "31aa38fa-7343-498c-8e17-9a7fd0c07b7b") : secret "model-serving-api-tls" not found
Jul 29 17:01:33.139192 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.139149    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e3914fc-3283-494d-9546-4ef0cffb78aa-cert\") pod \"odh-model-controller-75d9cdbddf-kl5hm\" (UID: \"3e3914fc-3283-494d-9546-4ef0cffb78aa\") " pod="kserve/odh-model-controller-75d9cdbddf-kl5hm"
Jul 29 17:01:33.146690 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.146618    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk5zg\" (UniqueName: \"kubernetes.io/projected/3e3914fc-3283-494d-9546-4ef0cffb78aa-kube-api-access-kk5zg\") pod \"odh-model-controller-75d9cdbddf-kl5hm\" (UID: \"3e3914fc-3283-494d-9546-4ef0cffb78aa\") " pod="kserve/odh-model-controller-75d9cdbddf-kl5hm"
Jul 29 17:01:33.146813 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.146798    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txqb7\" (UniqueName: \"kubernetes.io/projected/31aa38fa-7343-498c-8e17-9a7fd0c07b7b-kube-api-access-txqb7\") pod \"model-serving-api-7b954ffc95-k9vr4\" (UID: \"31aa38fa-7343-498c-8e17-9a7fd0c07b7b\") " pod="kserve/model-serving-api-7b954ffc95-k9vr4"
Jul 29 17:01:33.218076 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.218034    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-kl5hm"
Jul 29 17:01:33.360191 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.360138    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-kl5hm"]
Jul 29 17:01:33.361697 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:01:33.361673    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3914fc_3283_494d_9546_4ef0cffb78aa.slice/crio-f5c09564efa7f0f6f25feb02d0223c7fed8846f00aa7b813a0236b525bb9cdd1 WatchSource:0}: Error finding container f5c09564efa7f0f6f25feb02d0223c7fed8846f00aa7b813a0236b525bb9cdd1: Status 404 returned error can't find the container with id f5c09564efa7f0f6f25feb02d0223c7fed8846f00aa7b813a0236b525bb9cdd1
Jul 29 17:01:33.362914 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.362895    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 17:01:33.640768 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.640731    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31aa38fa-7343-498c-8e17-9a7fd0c07b7b-tls-certs\") pod \"model-serving-api-7b954ffc95-k9vr4\" (UID: \"31aa38fa-7343-498c-8e17-9a7fd0c07b7b\") " pod="kserve/model-serving-api-7b954ffc95-k9vr4"
Jul 29 17:01:33.643279 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.643248    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31aa38fa-7343-498c-8e17-9a7fd0c07b7b-tls-certs\") pod \"model-serving-api-7b954ffc95-k9vr4\" (UID: \"31aa38fa-7343-498c-8e17-9a7fd0c07b7b\") " pod="kserve/model-serving-api-7b954ffc95-k9vr4"
Jul 29 17:01:33.805301 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.805267    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-k9vr4"
Jul 29 17:01:33.969937 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:33.969904    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-k9vr4"]
Jul 29 17:01:33.973233 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:01:33.973198    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31aa38fa_7343_498c_8e17_9a7fd0c07b7b.slice/crio-3540f9d1da78ab4b60cc09e82773ca82bdbf07aa39587a3c37876a526fd78bd2 WatchSource:0}: Error finding container 3540f9d1da78ab4b60cc09e82773ca82bdbf07aa39587a3c37876a526fd78bd2: Status 404 returned error can't find the container with id 3540f9d1da78ab4b60cc09e82773ca82bdbf07aa39587a3c37876a526fd78bd2
Jul 29 17:01:34.014202 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:34.014133    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-k9vr4" event={"ID":"31aa38fa-7343-498c-8e17-9a7fd0c07b7b","Type":"ContainerStarted","Data":"3540f9d1da78ab4b60cc09e82773ca82bdbf07aa39587a3c37876a526fd78bd2"}
Jul 29 17:01:34.015962 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:34.015932    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-kl5hm" event={"ID":"3e3914fc-3283-494d-9546-4ef0cffb78aa","Type":"ContainerStarted","Data":"f5c09564efa7f0f6f25feb02d0223c7fed8846f00aa7b813a0236b525bb9cdd1"}
Jul 29 17:01:36.030730 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:36.030685    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-kl5hm" event={"ID":"3e3914fc-3283-494d-9546-4ef0cffb78aa","Type":"ContainerStarted","Data":"6557fd17852471735f65df9690f6f9cd3cfe39bf096df3976fa8c86c85df6c0b"}
Jul 29 17:01:36.031247 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:36.030769    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-kl5hm"
Jul 29 17:01:36.050507 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:36.050437    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-kl5hm" podStartSLOduration=1.471410052 podStartE2EDuration="4.050410814s" totalImagesPullingTime="2.579000762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:01:36.048566824 +0000 UTC m=+624.538086471" watchObservedRunningTime="2026-07-29 17:01:36.050410814 +0000 UTC m=+624.539930472"
Jul 29 17:01:39.052379 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:39.052338    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-k9vr4" event={"ID":"31aa38fa-7343-498c-8e17-9a7fd0c07b7b","Type":"ContainerStarted","Data":"ac3d427b3b899d6f1ec4b136176209193dd1db5f8be234bd14d820557506de00"}
Jul 29 17:01:39.052761 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:39.052424    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-k9vr4"
Jul 29 17:01:39.073960 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:39.073907    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-k9vr4" podStartSLOduration=3.068006355 podStartE2EDuration="7.073892568s" totalImagesPullingTime="4.005886213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:01:39.071417826 +0000 UTC m=+627.560937484" watchObservedRunningTime="2026-07-29 17:01:39.073892568 +0000 UTC m=+627.563412251"
Jul 29 17:01:47.041284 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:47.041250    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-kl5hm"
Jul 29 17:01:47.914101 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:47.914069    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-6kv5c"]
Jul 29 17:01:47.917515 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:47.917497    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6kv5c"
Jul 29 17:01:47.925489 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:47.925467    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-6kv5c"]
Jul 29 17:01:47.962988 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:47.962946    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hspwn\" (UniqueName: \"kubernetes.io/projected/71d092b8-2461-4a6c-8f00-54ac4f4b94c6-kube-api-access-hspwn\") pod \"s3-init-6kv5c\" (UID: \"71d092b8-2461-4a6c-8f00-54ac4f4b94c6\") " pod="kserve/s3-init-6kv5c"
Jul 29 17:01:48.064354 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:48.064312    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hspwn\" (UniqueName: \"kubernetes.io/projected/71d092b8-2461-4a6c-8f00-54ac4f4b94c6-kube-api-access-hspwn\") pod \"s3-init-6kv5c\" (UID: \"71d092b8-2461-4a6c-8f00-54ac4f4b94c6\") " pod="kserve/s3-init-6kv5c"
Jul 29 17:01:48.073426 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:48.073402    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hspwn\" (UniqueName: \"kubernetes.io/projected/71d092b8-2461-4a6c-8f00-54ac4f4b94c6-kube-api-access-hspwn\") pod \"s3-init-6kv5c\" (UID: \"71d092b8-2461-4a6c-8f00-54ac4f4b94c6\") " pod="kserve/s3-init-6kv5c"
Jul 29 17:01:48.227726 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:48.227632    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6kv5c"
Jul 29 17:01:48.383891 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:48.383863    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-6kv5c"]
Jul 29 17:01:49.110379 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:49.110310    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6kv5c" event={"ID":"71d092b8-2461-4a6c-8f00-54ac4f4b94c6","Type":"ContainerStarted","Data":"a5dd31d8ea2d11f196df19baf307af1e117b94fafc628c134c1b9dea916df38c"}
Jul 29 17:01:50.063115 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:50.063085    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-k9vr4"
Jul 29 17:01:53.138338 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:53.138305    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6kv5c" event={"ID":"71d092b8-2461-4a6c-8f00-54ac4f4b94c6","Type":"ContainerStarted","Data":"5944fe8aba896bc6b5fb9cbba9aadd9b4b586209ddab3810c44587066a7a872b"}
Jul 29 17:01:57.162482 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:57.162447    2570 generic.go:356] "Generic (PLEG): container finished" podID="71d092b8-2461-4a6c-8f00-54ac4f4b94c6" containerID="5944fe8aba896bc6b5fb9cbba9aadd9b4b586209ddab3810c44587066a7a872b" exitCode=0
Jul 29 17:01:57.162876 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:57.162521    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6kv5c" event={"ID":"71d092b8-2461-4a6c-8f00-54ac4f4b94c6","Type":"ContainerDied","Data":"5944fe8aba896bc6b5fb9cbba9aadd9b4b586209ddab3810c44587066a7a872b"}
Jul 29 17:01:58.314489 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:58.314462    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6kv5c"
Jul 29 17:01:58.356087 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:58.356053    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hspwn\" (UniqueName: \"kubernetes.io/projected/71d092b8-2461-4a6c-8f00-54ac4f4b94c6-kube-api-access-hspwn\") pod \"71d092b8-2461-4a6c-8f00-54ac4f4b94c6\" (UID: \"71d092b8-2461-4a6c-8f00-54ac4f4b94c6\") "
Jul 29 17:01:58.358213 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:58.358175    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d092b8-2461-4a6c-8f00-54ac4f4b94c6-kube-api-access-hspwn" (OuterVolumeSpecName: "kube-api-access-hspwn") pod "71d092b8-2461-4a6c-8f00-54ac4f4b94c6" (UID: "71d092b8-2461-4a6c-8f00-54ac4f4b94c6"). InnerVolumeSpecName "kube-api-access-hspwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:01:58.456780 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:58.456685    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hspwn\" (UniqueName: \"kubernetes.io/projected/71d092b8-2461-4a6c-8f00-54ac4f4b94c6-kube-api-access-hspwn\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:01:59.174361 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:59.174331    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6kv5c"
Jul 29 17:01:59.174535 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:59.174331    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6kv5c" event={"ID":"71d092b8-2461-4a6c-8f00-54ac4f4b94c6","Type":"ContainerDied","Data":"a5dd31d8ea2d11f196df19baf307af1e117b94fafc628c134c1b9dea916df38c"}
Jul 29 17:01:59.174535 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:01:59.174444    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5dd31d8ea2d11f196df19baf307af1e117b94fafc628c134c1b9dea916df38c"
Jul 29 17:02:22.013228 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.013157    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"]
Jul 29 17:02:22.013766 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.013589    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d092b8-2461-4a6c-8f00-54ac4f4b94c6" containerName="s3-init"
Jul 29 17:02:22.013766 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.013601    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d092b8-2461-4a6c-8f00-54ac4f4b94c6" containerName="s3-init"
Jul 29 17:02:22.013766 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.013671    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="71d092b8-2461-4a6c-8f00-54ac4f4b94c6" containerName="s3-init"
Jul 29 17:02:22.018997 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.018976    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.022124 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.022093    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-szzmj\""
Jul 29 17:02:22.022268 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.022134    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 17:02:22.023033 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.023014    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 17:02:22.023115 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.023099    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 29 17:02:22.039593 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.039557    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"]
Jul 29 17:02:22.076285 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.076244    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-dshm\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.076506 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.076307    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-model-cache\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.076506 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.076335    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.076506 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.076358    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm686\" (UniqueName: \"kubernetes.io/projected/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-kube-api-access-wm686\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.076506 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.076395    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-tls-certs\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.076506 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.076433    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.076506 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.076490    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-home\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.177477 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.177436    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-model-cache\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.177477 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.177474    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.177725 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.177499    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm686\" (UniqueName: \"kubernetes.io/projected/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-kube-api-access-wm686\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.177725 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.177525    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-tls-certs\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.177725 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.177557    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.177725 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.177613    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-home\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.177725 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.177674    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-dshm\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.177988 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.177864    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-model-cache\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.178047 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.178006    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-home\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.178047 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.178024    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.178249 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.178228    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.180048 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.180028    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-dshm\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.180154 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.180129    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-tls-certs\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.187070 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.187007    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm686\" (UniqueName: \"kubernetes.io/projected/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-kube-api-access-wm686\") pod \"scheduler-inline-config-test-kserve-5c479f4484-qzpz4\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.330120 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.330024    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:22.486854 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.486830    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"]
Jul 29 17:02:22.489343 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:02:22.489306    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fc9f68_ef58_4c32_89e3_0ff48e40e16b.slice/crio-fca0828a317c98bcbfa9feaff2afd1b1e516c7c4eea3f39304e925457edde489 WatchSource:0}: Error finding container fca0828a317c98bcbfa9feaff2afd1b1e516c7c4eea3f39304e925457edde489: Status 404 returned error can't find the container with id fca0828a317c98bcbfa9feaff2afd1b1e516c7c4eea3f39304e925457edde489
Jul 29 17:02:22.505349 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.505317    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"]
Jul 29 17:02:22.510672 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.510648    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.513644 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.513621    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-jwlw2\""
Jul 29 17:02:22.530971 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.530942    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"]
Jul 29 17:02:22.581270 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.581154    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.581270 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.581222    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.581270 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.581258    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.581498 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.581276    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.581498 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.581292    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n99tn\" (UniqueName: \"kubernetes.io/projected/5bb62692-0c9c-4865-89a2-e5b578c1caaa-kube-api-access-n99tn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.581498 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.581316    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.682434 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.682398    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.682434 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.682441    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.682689 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.682514    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.682689 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.682532    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.682689 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.682557    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n99tn\" (UniqueName: \"kubernetes.io/projected/5bb62692-0c9c-4865-89a2-e5b578c1caaa-kube-api-access-n99tn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.682689 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.682602    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.683028 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.682960    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.683028 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.682984    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.683206 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.683045    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.683206 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.683188    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.685357 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.685338    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.694240 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.694218    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n99tn\" (UniqueName: \"kubernetes.io/projected/5bb62692-0c9c-4865-89a2-e5b578c1caaa-kube-api-access-n99tn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.821896 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.821857    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:02:22.985213 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:22.985180    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"]
Jul 29 17:02:22.986085 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:02:22.986062    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb62692_0c9c_4865_89a2_e5b578c1caaa.slice/crio-4ee2b7ba7b41c32c3bab1642b225b2f8614af2bab45fc0c18f582650547bf887 WatchSource:0}: Error finding container 4ee2b7ba7b41c32c3bab1642b225b2f8614af2bab45fc0c18f582650547bf887: Status 404 returned error can't find the container with id 4ee2b7ba7b41c32c3bab1642b225b2f8614af2bab45fc0c18f582650547bf887
Jul 29 17:02:23.311454 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:23.311416    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4" event={"ID":"89fc9f68-ef58-4c32-89e3-0ff48e40e16b","Type":"ContainerStarted","Data":"fca0828a317c98bcbfa9feaff2afd1b1e516c7c4eea3f39304e925457edde489"}
Jul 29 17:02:23.312740 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:23.312716    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz" event={"ID":"5bb62692-0c9c-4865-89a2-e5b578c1caaa","Type":"ContainerStarted","Data":"4ee2b7ba7b41c32c3bab1642b225b2f8614af2bab45fc0c18f582650547bf887"}
Jul 29 17:02:27.341000 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:27.340905    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4" event={"ID":"89fc9f68-ef58-4c32-89e3-0ff48e40e16b","Type":"ContainerStarted","Data":"10e6e2f1b51a88f0ddb7c42a534a19dac534a018ed5e0e12a17827875eaf6fe8"}
Jul 29 17:02:27.342854 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:27.342822    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz" event={"ID":"5bb62692-0c9c-4865-89a2-e5b578c1caaa","Type":"ContainerStarted","Data":"cc3bea3de5964746f6478a0359ab672a63f8925bfb0288b7a15c2da51f6c2a9b"}
Jul 29 17:02:28.352335 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:28.352302    2570 generic.go:356] "Generic (PLEG): container finished" podID="5bb62692-0c9c-4865-89a2-e5b578c1caaa" containerID="cc3bea3de5964746f6478a0359ab672a63f8925bfb0288b7a15c2da51f6c2a9b" exitCode=0
Jul 29 17:02:28.352830 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:28.352372    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz" event={"ID":"5bb62692-0c9c-4865-89a2-e5b578c1caaa","Type":"ContainerDied","Data":"cc3bea3de5964746f6478a0359ab672a63f8925bfb0288b7a15c2da51f6c2a9b"}
Jul 29 17:02:30.376968 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:30.376908    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz" event={"ID":"5bb62692-0c9c-4865-89a2-e5b578c1caaa","Type":"ContainerStarted","Data":"0ae265f93ffcc3f4ff8fe8a6396b4ede6e66ba0cf3c4a2cdd9e973b76fa369ea"}
Jul 29 17:02:33.402017 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:33.401928    2570 generic.go:356] "Generic (PLEG): container finished" podID="89fc9f68-ef58-4c32-89e3-0ff48e40e16b" containerID="10e6e2f1b51a88f0ddb7c42a534a19dac534a018ed5e0e12a17827875eaf6fe8" exitCode=0
Jul 29 17:02:33.402511 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:33.402013    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4" event={"ID":"89fc9f68-ef58-4c32-89e3-0ff48e40e16b","Type":"ContainerDied","Data":"10e6e2f1b51a88f0ddb7c42a534a19dac534a018ed5e0e12a17827875eaf6fe8"}
Jul 29 17:02:35.419129 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:35.419085    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4" event={"ID":"89fc9f68-ef58-4c32-89e3-0ff48e40e16b","Type":"ContainerStarted","Data":"8de26069fbfe7278bbc80faf22ebec843c0c71b242e455a6da0f5857fa8f315c"}
Jul 29 17:02:35.440765 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:35.440704    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4" podStartSLOduration=3.291817336 podStartE2EDuration="14.440685829s" totalImagesPullingTime="5.852963538s" totalInitContainerRuntime="5.295904955s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:02:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:02:35.437801041 +0000 UTC m=+683.927320698" watchObservedRunningTime="2026-07-29 17:02:35.440685829 +0000 UTC m=+683.930205487"
Jul 29 17:02:42.330930 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:42.330886    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:42.331607 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:42.330957    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:42.346395 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:42.346359    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:02:42.480331 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:02:42.480281    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:03:00.581202 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:00.581141    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz" event={"ID":"5bb62692-0c9c-4865-89a2-e5b578c1caaa","Type":"ContainerStarted","Data":"59ec730b35b273dc12510015729f4a1ce32e93544f44d9d5caed753213ee5b52"}
Jul 29 17:03:00.581669 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:00.581349    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:03:00.584213 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:00.584189    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:03:00.607219 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:00.607152    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz" podStartSLOduration=2.151416844 podStartE2EDuration="38.607132842s" totalImagesPullingTime="35.744160848s" totalInitContainerRuntime="711.55515ms" isStatefulPod=false podCreationTimestamp="2026-07-29 17:02:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:03:00.604143568 +0000 UTC m=+709.093663263" watchObservedRunningTime="2026-07-29 17:03:00.607132842 +0000 UTC m=+709.096652500"
Jul 29 17:03:02.822377 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:02.822331    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:03:02.822777 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:02.822444    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:03:12.824308 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:12.824275    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:03:12.825372 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:12.825351    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:03:34.026851 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.026812    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"]
Jul 29 17:03:34.027394 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.027124    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz" podUID="5bb62692-0c9c-4865-89a2-e5b578c1caaa" containerName="main" containerID="cri-o://0ae265f93ffcc3f4ff8fe8a6396b4ede6e66ba0cf3c4a2cdd9e973b76fa369ea" gracePeriod=30
Jul 29 17:03:34.027394 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.027204    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz" podUID="5bb62692-0c9c-4865-89a2-e5b578c1caaa" containerName="tokenizer" containerID="cri-o://59ec730b35b273dc12510015729f4a1ce32e93544f44d9d5caed753213ee5b52" gracePeriod=30
Jul 29 17:03:34.030562 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.030534    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"]
Jul 29 17:03:34.030924 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.030794    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4" podUID="89fc9f68-ef58-4c32-89e3-0ff48e40e16b" containerName="main" containerID="cri-o://8de26069fbfe7278bbc80faf22ebec843c0c71b242e455a6da0f5857fa8f315c" gracePeriod=30
Jul 29 17:03:34.328331 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.328299    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:03:34.416229 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.416190    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-dshm\") pod \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") "
Jul 29 17:03:34.416415 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.416244    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-model-cache\") pod \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") "
Jul 29 17:03:34.416415 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.416285    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-tmp-dir\") pod \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") "
Jul 29 17:03:34.416415 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.416307    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm686\" (UniqueName: \"kubernetes.io/projected/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-kube-api-access-wm686\") pod \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") "
Jul 29 17:03:34.416415 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.416340    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-home\") pod \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") "
Jul 29 17:03:34.416415 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.416367    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-tls-certs\") pod \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") "
Jul 29 17:03:34.416415 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.416402    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-kserve-provision-location\") pod \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\" (UID: \"89fc9f68-ef58-4c32-89e3-0ff48e40e16b\") "
Jul 29 17:03:34.416742 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.416513    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-model-cache" (OuterVolumeSpecName: "model-cache") pod "89fc9f68-ef58-4c32-89e3-0ff48e40e16b" (UID: "89fc9f68-ef58-4c32-89e3-0ff48e40e16b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:03:34.416742 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.416529    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89fc9f68-ef58-4c32-89e3-0ff48e40e16b" (UID: "89fc9f68-ef58-4c32-89e3-0ff48e40e16b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:03:34.416742 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.416593    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-home" (OuterVolumeSpecName: "home") pod "89fc9f68-ef58-4c32-89e3-0ff48e40e16b" (UID: "89fc9f68-ef58-4c32-89e3-0ff48e40e16b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:03:34.416742 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.416697    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:03:34.416742 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.416718    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:03:34.416742 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.416733    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:03:34.418960 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.418930    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-dshm" (OuterVolumeSpecName: "dshm") pod "89fc9f68-ef58-4c32-89e3-0ff48e40e16b" (UID: "89fc9f68-ef58-4c32-89e3-0ff48e40e16b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:03:34.419066 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.419028    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89fc9f68-ef58-4c32-89e3-0ff48e40e16b" (UID: "89fc9f68-ef58-4c32-89e3-0ff48e40e16b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:03:34.419136 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.419118    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-kube-api-access-wm686" (OuterVolumeSpecName: "kube-api-access-wm686") pod "89fc9f68-ef58-4c32-89e3-0ff48e40e16b" (UID: "89fc9f68-ef58-4c32-89e3-0ff48e40e16b"). InnerVolumeSpecName "kube-api-access-wm686". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:03:34.471561 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.471496    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89fc9f68-ef58-4c32-89e3-0ff48e40e16b" (UID: "89fc9f68-ef58-4c32-89e3-0ff48e40e16b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:03:34.517252 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.517217    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:03:34.517252 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.517255    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:03:34.517469 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.517273    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:03:34.517469 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.517288    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm686\" (UniqueName: \"kubernetes.io/projected/89fc9f68-ef58-4c32-89e3-0ff48e40e16b-kube-api-access-wm686\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:03:34.767592 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.767545    2570 generic.go:356] "Generic (PLEG): container finished" podID="89fc9f68-ef58-4c32-89e3-0ff48e40e16b" containerID="8de26069fbfe7278bbc80faf22ebec843c0c71b242e455a6da0f5857fa8f315c" exitCode=0
Jul 29 17:03:34.767801 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.767617    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"
Jul 29 17:03:34.767801 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.767632    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4" event={"ID":"89fc9f68-ef58-4c32-89e3-0ff48e40e16b","Type":"ContainerDied","Data":"8de26069fbfe7278bbc80faf22ebec843c0c71b242e455a6da0f5857fa8f315c"}
Jul 29 17:03:34.767801 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.767695    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4" event={"ID":"89fc9f68-ef58-4c32-89e3-0ff48e40e16b","Type":"ContainerDied","Data":"fca0828a317c98bcbfa9feaff2afd1b1e516c7c4eea3f39304e925457edde489"}
Jul 29 17:03:34.767801 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.767725    2570 scope.go:117] "RemoveContainer" containerID="8de26069fbfe7278bbc80faf22ebec843c0c71b242e455a6da0f5857fa8f315c"
Jul 29 17:03:34.770244 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.770224    2570 generic.go:356] "Generic (PLEG): container finished" podID="5bb62692-0c9c-4865-89a2-e5b578c1caaa" containerID="0ae265f93ffcc3f4ff8fe8a6396b4ede6e66ba0cf3c4a2cdd9e973b76fa369ea" exitCode=0
Jul 29 17:03:34.770365 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.770293    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz" event={"ID":"5bb62692-0c9c-4865-89a2-e5b578c1caaa","Type":"ContainerDied","Data":"0ae265f93ffcc3f4ff8fe8a6396b4ede6e66ba0cf3c4a2cdd9e973b76fa369ea"}
Jul 29 17:03:34.777414 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.777395    2570 scope.go:117] "RemoveContainer" containerID="10e6e2f1b51a88f0ddb7c42a534a19dac534a018ed5e0e12a17827875eaf6fe8"
Jul 29 17:03:34.794378 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.794346    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"]
Jul 29 17:03:34.798139 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.798113    2570 scope.go:117] "RemoveContainer" containerID="8de26069fbfe7278bbc80faf22ebec843c0c71b242e455a6da0f5857fa8f315c"
Jul 29 17:03:34.798592 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:03:34.798569    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de26069fbfe7278bbc80faf22ebec843c0c71b242e455a6da0f5857fa8f315c\": container with ID starting with 8de26069fbfe7278bbc80faf22ebec843c0c71b242e455a6da0f5857fa8f315c not found: ID does not exist" containerID="8de26069fbfe7278bbc80faf22ebec843c0c71b242e455a6da0f5857fa8f315c"
Jul 29 17:03:34.798673 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.798601    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de26069fbfe7278bbc80faf22ebec843c0c71b242e455a6da0f5857fa8f315c"} err="failed to get container status \"8de26069fbfe7278bbc80faf22ebec843c0c71b242e455a6da0f5857fa8f315c\": rpc error: code = NotFound desc = could not find container \"8de26069fbfe7278bbc80faf22ebec843c0c71b242e455a6da0f5857fa8f315c\": container with ID starting with 8de26069fbfe7278bbc80faf22ebec843c0c71b242e455a6da0f5857fa8f315c not found: ID does not exist"
Jul 29 17:03:34.798673 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.798620    2570 scope.go:117] "RemoveContainer" containerID="10e6e2f1b51a88f0ddb7c42a534a19dac534a018ed5e0e12a17827875eaf6fe8"
Jul 29 17:03:34.798924 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:03:34.798902    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e6e2f1b51a88f0ddb7c42a534a19dac534a018ed5e0e12a17827875eaf6fe8\": container with ID starting with 10e6e2f1b51a88f0ddb7c42a534a19dac534a018ed5e0e12a17827875eaf6fe8 not found: ID does not exist" containerID="10e6e2f1b51a88f0ddb7c42a534a19dac534a018ed5e0e12a17827875eaf6fe8"
Jul 29 17:03:34.798976 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.798930    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e6e2f1b51a88f0ddb7c42a534a19dac534a018ed5e0e12a17827875eaf6fe8"} err="failed to get container status \"10e6e2f1b51a88f0ddb7c42a534a19dac534a018ed5e0e12a17827875eaf6fe8\": rpc error: code = NotFound desc = could not find container \"10e6e2f1b51a88f0ddb7c42a534a19dac534a018ed5e0e12a17827875eaf6fe8\": container with ID starting with 10e6e2f1b51a88f0ddb7c42a534a19dac534a018ed5e0e12a17827875eaf6fe8 not found: ID does not exist"
Jul 29 17:03:34.800797 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:34.800778    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5c479f4484-qzpz4"]
Jul 29 17:03:35.398761 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.398737    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:03:35.424103 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.424078    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tls-certs\") pod \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") "
Jul 29 17:03:35.424307 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.424127    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-tmp\") pod \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") "
Jul 29 17:03:35.424307 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.424181    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-uds\") pod \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") "
Jul 29 17:03:35.424307 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.424217    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n99tn\" (UniqueName: \"kubernetes.io/projected/5bb62692-0c9c-4865-89a2-e5b578c1caaa-kube-api-access-n99tn\") pod \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") "
Jul 29 17:03:35.424307 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.424250    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-cache\") pod \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") "
Jul 29 17:03:35.424307 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.424290    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-kserve-provision-location\") pod \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\" (UID: \"5bb62692-0c9c-4865-89a2-e5b578c1caaa\") "
Jul 29 17:03:35.424583 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.424547    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "5bb62692-0c9c-4865-89a2-e5b578c1caaa" (UID: "5bb62692-0c9c-4865-89a2-e5b578c1caaa"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:03:35.424728 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.424708    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "5bb62692-0c9c-4865-89a2-e5b578c1caaa" (UID: "5bb62692-0c9c-4865-89a2-e5b578c1caaa"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:03:35.424810 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.424787    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "5bb62692-0c9c-4865-89a2-e5b578c1caaa" (UID: "5bb62692-0c9c-4865-89a2-e5b578c1caaa"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:03:35.425382 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.425353    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5bb62692-0c9c-4865-89a2-e5b578c1caaa" (UID: "5bb62692-0c9c-4865-89a2-e5b578c1caaa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:03:35.426615 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.426586    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bb62692-0c9c-4865-89a2-e5b578c1caaa" (UID: "5bb62692-0c9c-4865-89a2-e5b578c1caaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:03:35.427391 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.427360    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb62692-0c9c-4865-89a2-e5b578c1caaa-kube-api-access-n99tn" (OuterVolumeSpecName: "kube-api-access-n99tn") pod "5bb62692-0c9c-4865-89a2-e5b578c1caaa" (UID: "5bb62692-0c9c-4865-89a2-e5b578c1caaa"). InnerVolumeSpecName "kube-api-access-n99tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:03:35.525433 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.525398    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-tmp\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:03:35.525433 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.525426    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-uds\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:03:35.525433 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.525435    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n99tn\" (UniqueName: \"kubernetes.io/projected/5bb62692-0c9c-4865-89a2-e5b578c1caaa-kube-api-access-n99tn\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:03:35.525676 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.525444    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tokenizer-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:03:35.525676 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.525454    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb62692-0c9c-4865-89a2-e5b578c1caaa-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:03:35.525676 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.525463    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb62692-0c9c-4865-89a2-e5b578c1caaa-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:03:35.778638 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.778545    2570 generic.go:356] "Generic (PLEG): container finished" podID="5bb62692-0c9c-4865-89a2-e5b578c1caaa" containerID="59ec730b35b273dc12510015729f4a1ce32e93544f44d9d5caed753213ee5b52" exitCode=0
Jul 29 17:03:35.778638 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.778598    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"
Jul 29 17:03:35.778898 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.778638    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz" event={"ID":"5bb62692-0c9c-4865-89a2-e5b578c1caaa","Type":"ContainerDied","Data":"59ec730b35b273dc12510015729f4a1ce32e93544f44d9d5caed753213ee5b52"}
Jul 29 17:03:35.778898 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.778700    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz" event={"ID":"5bb62692-0c9c-4865-89a2-e5b578c1caaa","Type":"ContainerDied","Data":"4ee2b7ba7b41c32c3bab1642b225b2f8614af2bab45fc0c18f582650547bf887"}
Jul 29 17:03:35.778898 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.778731    2570 scope.go:117] "RemoveContainer" containerID="59ec730b35b273dc12510015729f4a1ce32e93544f44d9d5caed753213ee5b52"
Jul 29 17:03:35.794279 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.794260    2570 scope.go:117] "RemoveContainer" containerID="0ae265f93ffcc3f4ff8fe8a6396b4ede6e66ba0cf3c4a2cdd9e973b76fa369ea"
Jul 29 17:03:35.802388 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.802370    2570 scope.go:117] "RemoveContainer" containerID="cc3bea3de5964746f6478a0359ab672a63f8925bfb0288b7a15c2da51f6c2a9b"
Jul 29 17:03:35.811058 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.811036    2570 scope.go:117] "RemoveContainer" containerID="59ec730b35b273dc12510015729f4a1ce32e93544f44d9d5caed753213ee5b52"
Jul 29 17:03:35.811364 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:03:35.811341    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ec730b35b273dc12510015729f4a1ce32e93544f44d9d5caed753213ee5b52\": container with ID starting with 59ec730b35b273dc12510015729f4a1ce32e93544f44d9d5caed753213ee5b52 not found: ID does not exist" containerID="59ec730b35b273dc12510015729f4a1ce32e93544f44d9d5caed753213ee5b52"
Jul 29 17:03:35.811450 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.811384    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ec730b35b273dc12510015729f4a1ce32e93544f44d9d5caed753213ee5b52"} err="failed to get container status \"59ec730b35b273dc12510015729f4a1ce32e93544f44d9d5caed753213ee5b52\": rpc error: code = NotFound desc = could not find container \"59ec730b35b273dc12510015729f4a1ce32e93544f44d9d5caed753213ee5b52\": container with ID starting with 59ec730b35b273dc12510015729f4a1ce32e93544f44d9d5caed753213ee5b52 not found: ID does not exist"
Jul 29 17:03:35.811450 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.811408    2570 scope.go:117] "RemoveContainer" containerID="0ae265f93ffcc3f4ff8fe8a6396b4ede6e66ba0cf3c4a2cdd9e973b76fa369ea"
Jul 29 17:03:35.811626 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:03:35.811609    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae265f93ffcc3f4ff8fe8a6396b4ede6e66ba0cf3c4a2cdd9e973b76fa369ea\": container with ID starting with 0ae265f93ffcc3f4ff8fe8a6396b4ede6e66ba0cf3c4a2cdd9e973b76fa369ea not found: ID does not exist" containerID="0ae265f93ffcc3f4ff8fe8a6396b4ede6e66ba0cf3c4a2cdd9e973b76fa369ea"
Jul 29 17:03:35.811689 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.811630    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae265f93ffcc3f4ff8fe8a6396b4ede6e66ba0cf3c4a2cdd9e973b76fa369ea"} err="failed to get container status \"0ae265f93ffcc3f4ff8fe8a6396b4ede6e66ba0cf3c4a2cdd9e973b76fa369ea\": rpc error: code = NotFound desc = could not find container \"0ae265f93ffcc3f4ff8fe8a6396b4ede6e66ba0cf3c4a2cdd9e973b76fa369ea\": container with ID starting with 0ae265f93ffcc3f4ff8fe8a6396b4ede6e66ba0cf3c4a2cdd9e973b76fa369ea not found: ID does not exist"
Jul 29 17:03:35.811689 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.811651    2570 scope.go:117] "RemoveContainer" containerID="cc3bea3de5964746f6478a0359ab672a63f8925bfb0288b7a15c2da51f6c2a9b"
Jul 29 17:03:35.811862 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:03:35.811847    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3bea3de5964746f6478a0359ab672a63f8925bfb0288b7a15c2da51f6c2a9b\": container with ID starting with cc3bea3de5964746f6478a0359ab672a63f8925bfb0288b7a15c2da51f6c2a9b not found: ID does not exist" containerID="cc3bea3de5964746f6478a0359ab672a63f8925bfb0288b7a15c2da51f6c2a9b"
Jul 29 17:03:35.811920 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.811865    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3bea3de5964746f6478a0359ab672a63f8925bfb0288b7a15c2da51f6c2a9b"} err="failed to get container status \"cc3bea3de5964746f6478a0359ab672a63f8925bfb0288b7a15c2da51f6c2a9b\": rpc error: code = NotFound desc = could not find container \"cc3bea3de5964746f6478a0359ab672a63f8925bfb0288b7a15c2da51f6c2a9b\": container with ID starting with cc3bea3de5964746f6478a0359ab672a63f8925bfb0288b7a15c2da51f6c2a9b not found: ID does not exist"
Jul 29 17:03:35.812118 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.812097    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"]
Jul 29 17:03:35.818310 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.818286    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-7cb79bw5sz"]
Jul 29 17:03:35.913551 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.913520    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bb62692-0c9c-4865-89a2-e5b578c1caaa" path="/var/lib/kubelet/pods/5bb62692-0c9c-4865-89a2-e5b578c1caaa/volumes"
Jul 29 17:03:35.914008 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:35.913994    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89fc9f68-ef58-4c32-89e3-0ff48e40e16b" path="/var/lib/kubelet/pods/89fc9f68-ef58-4c32-89e3-0ff48e40e16b/volumes"
Jul 29 17:03:42.972184 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.972117    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"]
Jul 29 17:03:42.972681 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.972658    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89fc9f68-ef58-4c32-89e3-0ff48e40e16b" containerName="storage-initializer"
Jul 29 17:03:42.972681 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.972676    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fc9f68-ef58-4c32-89e3-0ff48e40e16b" containerName="storage-initializer"
Jul 29 17:03:42.972769 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.972689    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89fc9f68-ef58-4c32-89e3-0ff48e40e16b" containerName="main"
Jul 29 17:03:42.972769 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.972694    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fc9f68-ef58-4c32-89e3-0ff48e40e16b" containerName="main"
Jul 29 17:03:42.972769 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.972705    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb62692-0c9c-4865-89a2-e5b578c1caaa" containerName="tokenizer"
Jul 29 17:03:42.972769 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.972712    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb62692-0c9c-4865-89a2-e5b578c1caaa" containerName="tokenizer"
Jul 29 17:03:42.972769 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.972723    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb62692-0c9c-4865-89a2-e5b578c1caaa" containerName="storage-initializer"
Jul 29 17:03:42.972769 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.972728    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb62692-0c9c-4865-89a2-e5b578c1caaa" containerName="storage-initializer"
Jul 29 17:03:42.972769 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.972733    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb62692-0c9c-4865-89a2-e5b578c1caaa" containerName="main"
Jul 29 17:03:42.972769 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.972738    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb62692-0c9c-4865-89a2-e5b578c1caaa" containerName="main"
Jul 29 17:03:42.973036 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.972789    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb62692-0c9c-4865-89a2-e5b578c1caaa" containerName="main"
Jul 29 17:03:42.973036 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.972797    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb62692-0c9c-4865-89a2-e5b578c1caaa" containerName="tokenizer"
Jul 29 17:03:42.973036 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.972804    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="89fc9f68-ef58-4c32-89e3-0ff48e40e16b" containerName="main"
Jul 29 17:03:42.979634 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.978963    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:42.982151 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.982126    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 17:03:42.983272 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.983030    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 17:03:42.983404 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.983333    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 29 17:03:42.983474 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.983409    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-szzmj\""
Jul 29 17:03:42.987334 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:42.987076    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"]
Jul 29 17:03:43.085345 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.085301    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.085534 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.085365    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-dshm\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.085534 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.085384    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rq7s\" (UniqueName: \"kubernetes.io/projected/de4ac348-05fb-41dd-ae2c-437bf8f262aa-kube-api-access-8rq7s\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.085534 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.085401    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-home\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.085534 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.085419    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-model-cache\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.085534 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.085454    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de4ac348-05fb-41dd-ae2c-437bf8f262aa-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.085534 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.085523    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.186221 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.186155    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de4ac348-05fb-41dd-ae2c-437bf8f262aa-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.186430 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.186234    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.186430 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.186261    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.186430 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.186298    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-dshm\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.186430 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.186316    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rq7s\" (UniqueName: \"kubernetes.io/projected/de4ac348-05fb-41dd-ae2c-437bf8f262aa-kube-api-access-8rq7s\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.186430 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.186336    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-home\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.186430 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.186375    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-model-cache\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.186795 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.186766    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.186931 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.186867    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-home\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.186931 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.186881    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-model-cache\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.186931 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.186838    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.188683 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.188660    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-dshm\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.188800 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.188787    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de4ac348-05fb-41dd-ae2c-437bf8f262aa-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.195386 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.195355    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rq7s\" (UniqueName: \"kubernetes.io/projected/de4ac348-05fb-41dd-ae2c-437bf8f262aa-kube-api-access-8rq7s\") pod \"scheduler-configmap-ref-test-kserve-96b574bdd-49nwj\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.291812 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.291769    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:43.325877 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.325844    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"]
Jul 29 17:03:43.330206 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.330178    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.332777 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.332751    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-lvpph\""
Jul 29 17:03:43.345373 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.345343    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"]
Jul 29 17:03:43.389212 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.389173    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.389705 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.389498    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltnlj\" (UniqueName: \"kubernetes.io/projected/59f5cdee-4903-4422-8e02-d02ac106f01c-kube-api-access-ltnlj\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.389705 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.389536    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.389705 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.389568    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.389705 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.389597    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.389705 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.389632    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f5cdee-4903-4422-8e02-d02ac106f01c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.453571 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.453543    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"]
Jul 29 17:03:43.455236 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:03:43.455210    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4ac348_05fb_41dd_ae2c_437bf8f262aa.slice/crio-d4bf246bc5bcd22af472ab2c8f13899c3a6e51569f5288d3b9d9f781d586a03b WatchSource:0}: Error finding container d4bf246bc5bcd22af472ab2c8f13899c3a6e51569f5288d3b9d9f781d586a03b: Status 404 returned error can't find the container with id d4bf246bc5bcd22af472ab2c8f13899c3a6e51569f5288d3b9d9f781d586a03b
Jul 29 17:03:43.490255 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.490226    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.490394 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.490310    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltnlj\" (UniqueName: \"kubernetes.io/projected/59f5cdee-4903-4422-8e02-d02ac106f01c-kube-api-access-ltnlj\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.490394 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.490344    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.490394 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.490374    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.490563 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.490403    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.490563 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.490442    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f5cdee-4903-4422-8e02-d02ac106f01c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.490654 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.490633    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.490698 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.490670    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.490847 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.490823    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.490905 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.490848    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.492923 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.492899    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f5cdee-4903-4422-8e02-d02ac106f01c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.498689 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.498668    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltnlj\" (UniqueName: \"kubernetes.io/projected/59f5cdee-4903-4422-8e02-d02ac106f01c-kube-api-access-ltnlj\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.641531 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.641431    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:43.806704 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.806675    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"]
Jul 29 17:03:43.808650 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:03:43.808606    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f5cdee_4903_4422_8e02_d02ac106f01c.slice/crio-b980d886e873ab72bb0f32104c4bdb73f800ab09c50e90b410f6ba08c7bbfc9b WatchSource:0}: Error finding container b980d886e873ab72bb0f32104c4bdb73f800ab09c50e90b410f6ba08c7bbfc9b: Status 404 returned error can't find the container with id b980d886e873ab72bb0f32104c4bdb73f800ab09c50e90b410f6ba08c7bbfc9b
Jul 29 17:03:43.823919 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.823876    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns" event={"ID":"59f5cdee-4903-4422-8e02-d02ac106f01c","Type":"ContainerStarted","Data":"b980d886e873ab72bb0f32104c4bdb73f800ab09c50e90b410f6ba08c7bbfc9b"}
Jul 29 17:03:43.825763 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.825733    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj" event={"ID":"de4ac348-05fb-41dd-ae2c-437bf8f262aa","Type":"ContainerStarted","Data":"feda814db29eca28d6505425c4b0a5d9fa048687d65d578fd99f1aa9feac3c22"}
Jul 29 17:03:43.825886 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:43.825789    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj" event={"ID":"de4ac348-05fb-41dd-ae2c-437bf8f262aa","Type":"ContainerStarted","Data":"d4bf246bc5bcd22af472ab2c8f13899c3a6e51569f5288d3b9d9f781d586a03b"}
Jul 29 17:03:44.835053 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:44.835016    2570 generic.go:356] "Generic (PLEG): container finished" podID="59f5cdee-4903-4422-8e02-d02ac106f01c" containerID="a0e39f12d114dc5473a2c4d8531a75e90ef9dcc20ace6543e5455e3714db6e3c" exitCode=0
Jul 29 17:03:44.835606 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:44.835108    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns" event={"ID":"59f5cdee-4903-4422-8e02-d02ac106f01c","Type":"ContainerDied","Data":"a0e39f12d114dc5473a2c4d8531a75e90ef9dcc20ace6543e5455e3714db6e3c"}
Jul 29 17:03:45.843253 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:45.843213    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns" event={"ID":"59f5cdee-4903-4422-8e02-d02ac106f01c","Type":"ContainerStarted","Data":"0747746d135df98b178a6aafbb6b154978b527be54bbe1335bc2ca07909ed014"}
Jul 29 17:03:45.843797 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:45.843282    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns" event={"ID":"59f5cdee-4903-4422-8e02-d02ac106f01c","Type":"ContainerStarted","Data":"7c32d1979d517661d7acd92dd4411d36879dd17a87932e72e633722476400127"}
Jul 29 17:03:45.843797 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:45.843428    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:45.870868 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:45.870802    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns" podStartSLOduration=2.083918145 podStartE2EDuration="2.870783431s" totalImagesPullingTime="0s" totalInitContainerRuntime="786.865286ms" isStatefulPod=false podCreationTimestamp="2026-07-29 17:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:03:45.869068308 +0000 UTC m=+754.358588231" watchObservedRunningTime="2026-07-29 17:03:45.870783431 +0000 UTC m=+754.360303152"
Jul 29 17:03:48.862555 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:48.862517    2570 generic.go:356] "Generic (PLEG): container finished" podID="de4ac348-05fb-41dd-ae2c-437bf8f262aa" containerID="feda814db29eca28d6505425c4b0a5d9fa048687d65d578fd99f1aa9feac3c22" exitCode=0
Jul 29 17:03:48.862555 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:48.862561    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj" event={"ID":"de4ac348-05fb-41dd-ae2c-437bf8f262aa","Type":"ContainerDied","Data":"feda814db29eca28d6505425c4b0a5d9fa048687d65d578fd99f1aa9feac3c22"}
Jul 29 17:03:49.870210 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:49.870175    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj" event={"ID":"de4ac348-05fb-41dd-ae2c-437bf8f262aa","Type":"ContainerStarted","Data":"c00d15c351c17ed1a3b7ead1f8103b9f2d8b6ee851477dbf14e3ee4c73c82976"}
Jul 29 17:03:49.901203 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:49.901136    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj" podStartSLOduration=3.467177273 podStartE2EDuration="7.901115811s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.433938538s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:03:49.899930988 +0000 UTC m=+758.389450645" watchObservedRunningTime="2026-07-29 17:03:49.901115811 +0000 UTC m=+758.390635468"
Jul 29 17:03:53.292744 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:53.292707    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:53.293200 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:53.292773    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:53.305647 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:53.305617    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:03:53.641839 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:53.641740    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:53.641839 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:53.641813    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:53.644824 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:53.644795    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:53.893866 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:53.893765    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:03:53.904307 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:03:53.904272    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:04:14.899696 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:14.899663    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:04:48.637387 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.637303    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"]
Jul 29 17:04:48.641905 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.641879    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.645752 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.645723    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 29 17:04:48.645920 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.645834    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-wxllg\""
Jul 29 17:04:48.654848 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.654820    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"]
Jul 29 17:04:48.774833 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.774791    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.775020 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.774846    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8wvn\" (UniqueName: \"kubernetes.io/projected/137d69a8-afc1-4dfb-8de5-7e120a29f16f-kube-api-access-n8wvn\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.775020 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.774932    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.775020 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.774988    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.775130 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.775039    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.775130 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.775063    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.876277 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.876233    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.876277 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.876280    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.876524 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.876315    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.876524 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.876342    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8wvn\" (UniqueName: \"kubernetes.io/projected/137d69a8-afc1-4dfb-8de5-7e120a29f16f-kube-api-access-n8wvn\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.876524 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.876377    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.876524 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.876433    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.876806 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.876781    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.876875 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.876806    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.876875 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.876844    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.876968 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.876938    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.878764 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.878741    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.885150 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.885122    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8wvn\" (UniqueName: \"kubernetes.io/projected/137d69a8-afc1-4dfb-8de5-7e120a29f16f-kube-api-access-n8wvn\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:48.953545 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:48.953444    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:49.103638 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:49.103607    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"]
Jul 29 17:04:49.104746 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:04:49.104714    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137d69a8_afc1_4dfb_8de5_7e120a29f16f.slice/crio-f55ecc88452903591f28d01001be4f2975e150e1f661f815a76f99855eae7ed7 WatchSource:0}: Error finding container f55ecc88452903591f28d01001be4f2975e150e1f661f815a76f99855eae7ed7: Status 404 returned error can't find the container with id f55ecc88452903591f28d01001be4f2975e150e1f661f815a76f99855eae7ed7
Jul 29 17:04:49.193189 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:49.193136    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h" event={"ID":"137d69a8-afc1-4dfb-8de5-7e120a29f16f","Type":"ContainerStarted","Data":"d8a7d741d31bff7f8e302cac7883b80727245e90865358a06d8fd47fa22d6188"}
Jul 29 17:04:49.193389 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:49.193201    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h" event={"ID":"137d69a8-afc1-4dfb-8de5-7e120a29f16f","Type":"ContainerStarted","Data":"f55ecc88452903591f28d01001be4f2975e150e1f661f815a76f99855eae7ed7"}
Jul 29 17:04:50.084039 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.083996    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"]
Jul 29 17:04:50.084525 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.084269    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj" podUID="de4ac348-05fb-41dd-ae2c-437bf8f262aa" containerName="main" containerID="cri-o://c00d15c351c17ed1a3b7ead1f8103b9f2d8b6ee851477dbf14e3ee4c73c82976" gracePeriod=30
Jul 29 17:04:50.087758 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.087727    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"]
Jul 29 17:04:50.088237 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.088142    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns" podUID="59f5cdee-4903-4422-8e02-d02ac106f01c" containerName="tokenizer" containerID="cri-o://0747746d135df98b178a6aafbb6b154978b527be54bbe1335bc2ca07909ed014" gracePeriod=30
Jul 29 17:04:50.088402 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.088122    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns" podUID="59f5cdee-4903-4422-8e02-d02ac106f01c" containerName="main" containerID="cri-o://7c32d1979d517661d7acd92dd4411d36879dd17a87932e72e633722476400127" gracePeriod=30
Jul 29 17:04:50.202103 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.202079    2570 generic.go:356] "Generic (PLEG): container finished" podID="137d69a8-afc1-4dfb-8de5-7e120a29f16f" containerID="d8a7d741d31bff7f8e302cac7883b80727245e90865358a06d8fd47fa22d6188" exitCode=0
Jul 29 17:04:50.202238 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.202212    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h" event={"ID":"137d69a8-afc1-4dfb-8de5-7e120a29f16f","Type":"ContainerDied","Data":"d8a7d741d31bff7f8e302cac7883b80727245e90865358a06d8fd47fa22d6188"}
Jul 29 17:04:50.359272 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.359241    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:04:50.495417 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.495315    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-model-cache\") pod \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") "
Jul 29 17:04:50.495417 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.495395    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rq7s\" (UniqueName: \"kubernetes.io/projected/de4ac348-05fb-41dd-ae2c-437bf8f262aa-kube-api-access-8rq7s\") pod \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") "
Jul 29 17:04:50.495657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.495430    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-tmp-dir\") pod \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") "
Jul 29 17:04:50.495657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.495458    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-kserve-provision-location\") pod \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") "
Jul 29 17:04:50.495657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.495492    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de4ac348-05fb-41dd-ae2c-437bf8f262aa-tls-certs\") pod \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") "
Jul 29 17:04:50.495657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.495628    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "de4ac348-05fb-41dd-ae2c-437bf8f262aa" (UID: "de4ac348-05fb-41dd-ae2c-437bf8f262aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:04:50.495858 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.495711    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-home\") pod \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") "
Jul 29 17:04:50.495858 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.495729    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de4ac348-05fb-41dd-ae2c-437bf8f262aa" (UID: "de4ac348-05fb-41dd-ae2c-437bf8f262aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:04:50.495858 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.495785    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-dshm\") pod \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\" (UID: \"de4ac348-05fb-41dd-ae2c-437bf8f262aa\") "
Jul 29 17:04:50.495998 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.495918    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-home" (OuterVolumeSpecName: "home") pod "de4ac348-05fb-41dd-ae2c-437bf8f262aa" (UID: "de4ac348-05fb-41dd-ae2c-437bf8f262aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:04:50.496080 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.496062    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:04:50.496138 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.496087    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:04:50.496138 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.496100    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:04:50.497835 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.497807    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4ac348-05fb-41dd-ae2c-437bf8f262aa-kube-api-access-8rq7s" (OuterVolumeSpecName: "kube-api-access-8rq7s") pod "de4ac348-05fb-41dd-ae2c-437bf8f262aa" (UID: "de4ac348-05fb-41dd-ae2c-437bf8f262aa"). InnerVolumeSpecName "kube-api-access-8rq7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:04:50.498050 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.498031    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-dshm" (OuterVolumeSpecName: "dshm") pod "de4ac348-05fb-41dd-ae2c-437bf8f262aa" (UID: "de4ac348-05fb-41dd-ae2c-437bf8f262aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:04:50.498050 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.498039    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de4ac348-05fb-41dd-ae2c-437bf8f262aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de4ac348-05fb-41dd-ae2c-437bf8f262aa" (UID: "de4ac348-05fb-41dd-ae2c-437bf8f262aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:04:50.569044 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.569002    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de4ac348-05fb-41dd-ae2c-437bf8f262aa" (UID: "de4ac348-05fb-41dd-ae2c-437bf8f262aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:04:50.597356 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.597323    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:04:50.597356 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.597354    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de4ac348-05fb-41dd-ae2c-437bf8f262aa-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:04:50.597356 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.597364    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de4ac348-05fb-41dd-ae2c-437bf8f262aa-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:04:50.597577 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:50.597372    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rq7s\" (UniqueName: \"kubernetes.io/projected/de4ac348-05fb-41dd-ae2c-437bf8f262aa-kube-api-access-8rq7s\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:04:51.211712 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.211618    2570 generic.go:356] "Generic (PLEG): container finished" podID="59f5cdee-4903-4422-8e02-d02ac106f01c" containerID="7c32d1979d517661d7acd92dd4411d36879dd17a87932e72e633722476400127" exitCode=0
Jul 29 17:04:51.212200 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.211701    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns" event={"ID":"59f5cdee-4903-4422-8e02-d02ac106f01c","Type":"ContainerDied","Data":"7c32d1979d517661d7acd92dd4411d36879dd17a87932e72e633722476400127"}
Jul 29 17:04:51.213509 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.213482    2570 generic.go:356] "Generic (PLEG): container finished" podID="de4ac348-05fb-41dd-ae2c-437bf8f262aa" containerID="c00d15c351c17ed1a3b7ead1f8103b9f2d8b6ee851477dbf14e3ee4c73c82976" exitCode=0
Jul 29 17:04:51.213639 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.213534    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"
Jul 29 17:04:51.213639 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.213558    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj" event={"ID":"de4ac348-05fb-41dd-ae2c-437bf8f262aa","Type":"ContainerDied","Data":"c00d15c351c17ed1a3b7ead1f8103b9f2d8b6ee851477dbf14e3ee4c73c82976"}
Jul 29 17:04:51.213639 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.213603    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj" event={"ID":"de4ac348-05fb-41dd-ae2c-437bf8f262aa","Type":"ContainerDied","Data":"d4bf246bc5bcd22af472ab2c8f13899c3a6e51569f5288d3b9d9f781d586a03b"}
Jul 29 17:04:51.213639 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.213633    2570 scope.go:117] "RemoveContainer" containerID="c00d15c351c17ed1a3b7ead1f8103b9f2d8b6ee851477dbf14e3ee4c73c82976"
Jul 29 17:04:51.216423 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.216396    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h" event={"ID":"137d69a8-afc1-4dfb-8de5-7e120a29f16f","Type":"ContainerStarted","Data":"1d1d92bbf5907526934301cdc462fc131649eab05e866ebdd4a577eba7c9959e"}
Jul 29 17:04:51.216565 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.216453    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h" event={"ID":"137d69a8-afc1-4dfb-8de5-7e120a29f16f","Type":"ContainerStarted","Data":"e42894395ac0d65c2f36b10f8ad080ca7a14b999c5022e1dd8c9066f74fa2be6"}
Jul 29 17:04:51.216565 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.216484    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:51.224445 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.224429    2570 scope.go:117] "RemoveContainer" containerID="feda814db29eca28d6505425c4b0a5d9fa048687d65d578fd99f1aa9feac3c22"
Jul 29 17:04:51.237870 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.237847    2570 scope.go:117] "RemoveContainer" containerID="c00d15c351c17ed1a3b7ead1f8103b9f2d8b6ee851477dbf14e3ee4c73c82976"
Jul 29 17:04:51.238232 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:04:51.238204    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00d15c351c17ed1a3b7ead1f8103b9f2d8b6ee851477dbf14e3ee4c73c82976\": container with ID starting with c00d15c351c17ed1a3b7ead1f8103b9f2d8b6ee851477dbf14e3ee4c73c82976 not found: ID does not exist" containerID="c00d15c351c17ed1a3b7ead1f8103b9f2d8b6ee851477dbf14e3ee4c73c82976"
Jul 29 17:04:51.238336 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.238241    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00d15c351c17ed1a3b7ead1f8103b9f2d8b6ee851477dbf14e3ee4c73c82976"} err="failed to get container status \"c00d15c351c17ed1a3b7ead1f8103b9f2d8b6ee851477dbf14e3ee4c73c82976\": rpc error: code = NotFound desc = could not find container \"c00d15c351c17ed1a3b7ead1f8103b9f2d8b6ee851477dbf14e3ee4c73c82976\": container with ID starting with c00d15c351c17ed1a3b7ead1f8103b9f2d8b6ee851477dbf14e3ee4c73c82976 not found: ID does not exist"
Jul 29 17:04:51.238336 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.238259    2570 scope.go:117] "RemoveContainer" containerID="feda814db29eca28d6505425c4b0a5d9fa048687d65d578fd99f1aa9feac3c22"
Jul 29 17:04:51.238592 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:04:51.238569    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feda814db29eca28d6505425c4b0a5d9fa048687d65d578fd99f1aa9feac3c22\": container with ID starting with feda814db29eca28d6505425c4b0a5d9fa048687d65d578fd99f1aa9feac3c22 not found: ID does not exist" containerID="feda814db29eca28d6505425c4b0a5d9fa048687d65d578fd99f1aa9feac3c22"
Jul 29 17:04:51.238652 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.238598    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feda814db29eca28d6505425c4b0a5d9fa048687d65d578fd99f1aa9feac3c22"} err="failed to get container status \"feda814db29eca28d6505425c4b0a5d9fa048687d65d578fd99f1aa9feac3c22\": rpc error: code = NotFound desc = could not find container \"feda814db29eca28d6505425c4b0a5d9fa048687d65d578fd99f1aa9feac3c22\": container with ID starting with feda814db29eca28d6505425c4b0a5d9fa048687d65d578fd99f1aa9feac3c22 not found: ID does not exist"
Jul 29 17:04:51.241251 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.241202    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h" podStartSLOduration=2.543810225 podStartE2EDuration="3.241185377s" totalImagesPullingTime="0s" totalInitContainerRuntime="697.375152ms" isStatefulPod=false podCreationTimestamp="2026-07-29 17:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:04:51.240202175 +0000 UTC m=+819.729721831" watchObservedRunningTime="2026-07-29 17:04:51.241185377 +0000 UTC m=+819.730705029"
Jul 29 17:04:51.254726 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.254690    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"]
Jul 29 17:04:51.258907 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.258885    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-96b574bdd-49nwj"]
Jul 29 17:04:51.549393 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.549370    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:04:51.610441 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.610410    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-kserve-provision-location\") pod \"59f5cdee-4903-4422-8e02-d02ac106f01c\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") "
Jul 29 17:04:51.610696 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.610682    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-cache\") pod \"59f5cdee-4903-4422-8e02-d02ac106f01c\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") "
Jul 29 17:04:51.610823 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.610810    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltnlj\" (UniqueName: \"kubernetes.io/projected/59f5cdee-4903-4422-8e02-d02ac106f01c-kube-api-access-ltnlj\") pod \"59f5cdee-4903-4422-8e02-d02ac106f01c\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") "
Jul 29 17:04:51.610916 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.610906    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f5cdee-4903-4422-8e02-d02ac106f01c-tls-certs\") pod \"59f5cdee-4903-4422-8e02-d02ac106f01c\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") "
Jul 29 17:04:51.611045 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.611035    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-uds\") pod \"59f5cdee-4903-4422-8e02-d02ac106f01c\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") "
Jul 29 17:04:51.611144 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.611133    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-tmp\") pod \"59f5cdee-4903-4422-8e02-d02ac106f01c\" (UID: \"59f5cdee-4903-4422-8e02-d02ac106f01c\") "
Jul 29 17:04:51.611771 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.611747    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "59f5cdee-4903-4422-8e02-d02ac106f01c" (UID: "59f5cdee-4903-4422-8e02-d02ac106f01c"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:04:51.612583 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.612560    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59f5cdee-4903-4422-8e02-d02ac106f01c" (UID: "59f5cdee-4903-4422-8e02-d02ac106f01c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:04:51.613155 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.613139    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "59f5cdee-4903-4422-8e02-d02ac106f01c" (UID: "59f5cdee-4903-4422-8e02-d02ac106f01c"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:04:51.613749 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.613719    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "59f5cdee-4903-4422-8e02-d02ac106f01c" (UID: "59f5cdee-4903-4422-8e02-d02ac106f01c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:04:51.629326 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.629295    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59f5cdee-4903-4422-8e02-d02ac106f01c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59f5cdee-4903-4422-8e02-d02ac106f01c" (UID: "59f5cdee-4903-4422-8e02-d02ac106f01c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:04:51.632317 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.632295    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f5cdee-4903-4422-8e02-d02ac106f01c-kube-api-access-ltnlj" (OuterVolumeSpecName: "kube-api-access-ltnlj") pod "59f5cdee-4903-4422-8e02-d02ac106f01c" (UID: "59f5cdee-4903-4422-8e02-d02ac106f01c"). InnerVolumeSpecName "kube-api-access-ltnlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:04:51.712455 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.712423    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:04:51.712455 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.712451    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:04:51.712455 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.712461    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltnlj\" (UniqueName: \"kubernetes.io/projected/59f5cdee-4903-4422-8e02-d02ac106f01c-kube-api-access-ltnlj\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:04:51.712692 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.712471    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f5cdee-4903-4422-8e02-d02ac106f01c-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:04:51.712692 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.712479    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-uds\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:04:51.712692 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.712487    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/59f5cdee-4903-4422-8e02-d02ac106f01c-tokenizer-tmp\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:04:51.920425 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:51.920327    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de4ac348-05fb-41dd-ae2c-437bf8f262aa" path="/var/lib/kubelet/pods/de4ac348-05fb-41dd-ae2c-437bf8f262aa/volumes"
Jul 29 17:04:52.224659 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.224567    2570 generic.go:356] "Generic (PLEG): container finished" podID="59f5cdee-4903-4422-8e02-d02ac106f01c" containerID="0747746d135df98b178a6aafbb6b154978b527be54bbe1335bc2ca07909ed014" exitCode=0
Jul 29 17:04:52.224659 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.224637    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"
Jul 29 17:04:52.225188 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.224673    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns" event={"ID":"59f5cdee-4903-4422-8e02-d02ac106f01c","Type":"ContainerDied","Data":"0747746d135df98b178a6aafbb6b154978b527be54bbe1335bc2ca07909ed014"}
Jul 29 17:04:52.225188 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.224741    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns" event={"ID":"59f5cdee-4903-4422-8e02-d02ac106f01c","Type":"ContainerDied","Data":"b980d886e873ab72bb0f32104c4bdb73f800ab09c50e90b410f6ba08c7bbfc9b"}
Jul 29 17:04:52.225188 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.224775    2570 scope.go:117] "RemoveContainer" containerID="0747746d135df98b178a6aafbb6b154978b527be54bbe1335bc2ca07909ed014"
Jul 29 17:04:52.233850 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.233827    2570 scope.go:117] "RemoveContainer" containerID="7c32d1979d517661d7acd92dd4411d36879dd17a87932e72e633722476400127"
Jul 29 17:04:52.242515 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.242494    2570 scope.go:117] "RemoveContainer" containerID="a0e39f12d114dc5473a2c4d8531a75e90ef9dcc20ace6543e5455e3714db6e3c"
Jul 29 17:04:52.251911 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.251885    2570 scope.go:117] "RemoveContainer" containerID="0747746d135df98b178a6aafbb6b154978b527be54bbe1335bc2ca07909ed014"
Jul 29 17:04:52.252347 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:04:52.252264    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0747746d135df98b178a6aafbb6b154978b527be54bbe1335bc2ca07909ed014\": container with ID starting with 0747746d135df98b178a6aafbb6b154978b527be54bbe1335bc2ca07909ed014 not found: ID does not exist" containerID="0747746d135df98b178a6aafbb6b154978b527be54bbe1335bc2ca07909ed014"
Jul 29 17:04:52.252347 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.252280    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"]
Jul 29 17:04:52.252347 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.252301    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0747746d135df98b178a6aafbb6b154978b527be54bbe1335bc2ca07909ed014"} err="failed to get container status \"0747746d135df98b178a6aafbb6b154978b527be54bbe1335bc2ca07909ed014\": rpc error: code = NotFound desc = could not find container \"0747746d135df98b178a6aafbb6b154978b527be54bbe1335bc2ca07909ed014\": container with ID starting with 0747746d135df98b178a6aafbb6b154978b527be54bbe1335bc2ca07909ed014 not found: ID does not exist"
Jul 29 17:04:52.252347 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.252328    2570 scope.go:117] "RemoveContainer" containerID="7c32d1979d517661d7acd92dd4411d36879dd17a87932e72e633722476400127"
Jul 29 17:04:52.252630 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:04:52.252605    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c32d1979d517661d7acd92dd4411d36879dd17a87932e72e633722476400127\": container with ID starting with 7c32d1979d517661d7acd92dd4411d36879dd17a87932e72e633722476400127 not found: ID does not exist" containerID="7c32d1979d517661d7acd92dd4411d36879dd17a87932e72e633722476400127"
Jul 29 17:04:52.252698 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.252642    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c32d1979d517661d7acd92dd4411d36879dd17a87932e72e633722476400127"} err="failed to get container status \"7c32d1979d517661d7acd92dd4411d36879dd17a87932e72e633722476400127\": rpc error: code = NotFound desc = could not find container \"7c32d1979d517661d7acd92dd4411d36879dd17a87932e72e633722476400127\": container with ID starting with 7c32d1979d517661d7acd92dd4411d36879dd17a87932e72e633722476400127 not found: ID does not exist"
Jul 29 17:04:52.252698 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.252666    2570 scope.go:117] "RemoveContainer" containerID="a0e39f12d114dc5473a2c4d8531a75e90ef9dcc20ace6543e5455e3714db6e3c"
Jul 29 17:04:52.252995 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:04:52.252966    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e39f12d114dc5473a2c4d8531a75e90ef9dcc20ace6543e5455e3714db6e3c\": container with ID starting with a0e39f12d114dc5473a2c4d8531a75e90ef9dcc20ace6543e5455e3714db6e3c not found: ID does not exist" containerID="a0e39f12d114dc5473a2c4d8531a75e90ef9dcc20ace6543e5455e3714db6e3c"
Jul 29 17:04:52.253102 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.252997    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e39f12d114dc5473a2c4d8531a75e90ef9dcc20ace6543e5455e3714db6e3c"} err="failed to get container status \"a0e39f12d114dc5473a2c4d8531a75e90ef9dcc20ace6543e5455e3714db6e3c\": rpc error: code = NotFound desc = could not find container \"a0e39f12d114dc5473a2c4d8531a75e90ef9dcc20ace6543e5455e3714db6e3c\": container with ID starting with a0e39f12d114dc5473a2c4d8531a75e90ef9dcc20ace6543e5455e3714db6e3c not found: ID does not exist"
Jul 29 17:04:52.254825 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:52.254802    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7f5455lfns"]
Jul 29 17:04:53.917654 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:53.917612    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59f5cdee-4903-4422-8e02-d02ac106f01c" path="/var/lib/kubelet/pods/59f5cdee-4903-4422-8e02-d02ac106f01c/volumes"
Jul 29 17:04:58.953685 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:58.953637    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:58.954080 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:58.953722    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:58.956645 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:58.956618    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:04:59.266425 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:04:59.266398    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:05:20.272838 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:20.272799    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:05:21.343185 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:21.343141    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"]
Jul 29 17:05:21.343656 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:21.343403    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h" podUID="137d69a8-afc1-4dfb-8de5-7e120a29f16f" containerName="main" containerID="cri-o://e42894395ac0d65c2f36b10f8ad080ca7a14b999c5022e1dd8c9066f74fa2be6" gracePeriod=30
Jul 29 17:05:21.343656 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:21.343437    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h" podUID="137d69a8-afc1-4dfb-8de5-7e120a29f16f" containerName="tokenizer" containerID="cri-o://1d1d92bbf5907526934301cdc462fc131649eab05e866ebdd4a577eba7c9959e" gracePeriod=30
Jul 29 17:05:22.402556 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.402517    2570 generic.go:356] "Generic (PLEG): container finished" podID="137d69a8-afc1-4dfb-8de5-7e120a29f16f" containerID="e42894395ac0d65c2f36b10f8ad080ca7a14b999c5022e1dd8c9066f74fa2be6" exitCode=0
Jul 29 17:05:22.402963 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.402591    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h" event={"ID":"137d69a8-afc1-4dfb-8de5-7e120a29f16f","Type":"ContainerDied","Data":"e42894395ac0d65c2f36b10f8ad080ca7a14b999c5022e1dd8c9066f74fa2be6"}
Jul 29 17:05:22.709888 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.709864    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:05:22.778044 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.778015    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-tmp\") pod \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") "
Jul 29 17:05:22.778044 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.778045    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-uds\") pod \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") "
Jul 29 17:05:22.778317 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.778088    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-cache\") pod \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") "
Jul 29 17:05:22.778317 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.778113    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tls-certs\") pod \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") "
Jul 29 17:05:22.778317 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.778129    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8wvn\" (UniqueName: \"kubernetes.io/projected/137d69a8-afc1-4dfb-8de5-7e120a29f16f-kube-api-access-n8wvn\") pod \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") "
Jul 29 17:05:22.778317 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.778220    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-kserve-provision-location\") pod \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\" (UID: \"137d69a8-afc1-4dfb-8de5-7e120a29f16f\") "
Jul 29 17:05:22.778525 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.778328    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "137d69a8-afc1-4dfb-8de5-7e120a29f16f" (UID: "137d69a8-afc1-4dfb-8de5-7e120a29f16f"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:22.778525 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.778417    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "137d69a8-afc1-4dfb-8de5-7e120a29f16f" (UID: "137d69a8-afc1-4dfb-8de5-7e120a29f16f"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:22.778525 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.778428    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "137d69a8-afc1-4dfb-8de5-7e120a29f16f" (UID: "137d69a8-afc1-4dfb-8de5-7e120a29f16f"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:22.778525 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.778501    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-uds\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:22.778525 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.778521    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:22.778929 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.778908    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "137d69a8-afc1-4dfb-8de5-7e120a29f16f" (UID: "137d69a8-afc1-4dfb-8de5-7e120a29f16f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:22.780279 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.780258    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137d69a8-afc1-4dfb-8de5-7e120a29f16f-kube-api-access-n8wvn" (OuterVolumeSpecName: "kube-api-access-n8wvn") pod "137d69a8-afc1-4dfb-8de5-7e120a29f16f" (UID: "137d69a8-afc1-4dfb-8de5-7e120a29f16f"). InnerVolumeSpecName "kube-api-access-n8wvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:05:22.780369 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.780346    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "137d69a8-afc1-4dfb-8de5-7e120a29f16f" (UID: "137d69a8-afc1-4dfb-8de5-7e120a29f16f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:05:22.879085 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.879044    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:22.879085 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.879076    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tokenizer-tmp\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:22.879085 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.879090    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/137d69a8-afc1-4dfb-8de5-7e120a29f16f-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:22.879375 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:22.879103    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8wvn\" (UniqueName: \"kubernetes.io/projected/137d69a8-afc1-4dfb-8de5-7e120a29f16f-kube-api-access-n8wvn\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:23.413028 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.412993    2570 generic.go:356] "Generic (PLEG): container finished" podID="137d69a8-afc1-4dfb-8de5-7e120a29f16f" containerID="1d1d92bbf5907526934301cdc462fc131649eab05e866ebdd4a577eba7c9959e" exitCode=0
Jul 29 17:05:23.413510 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.413065    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"
Jul 29 17:05:23.413510 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.413077    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h" event={"ID":"137d69a8-afc1-4dfb-8de5-7e120a29f16f","Type":"ContainerDied","Data":"1d1d92bbf5907526934301cdc462fc131649eab05e866ebdd4a577eba7c9959e"}
Jul 29 17:05:23.413510 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.413150    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h" event={"ID":"137d69a8-afc1-4dfb-8de5-7e120a29f16f","Type":"ContainerDied","Data":"f55ecc88452903591f28d01001be4f2975e150e1f661f815a76f99855eae7ed7"}
Jul 29 17:05:23.413510 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.413201    2570 scope.go:117] "RemoveContainer" containerID="1d1d92bbf5907526934301cdc462fc131649eab05e866ebdd4a577eba7c9959e"
Jul 29 17:05:23.422592 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.422575    2570 scope.go:117] "RemoveContainer" containerID="e42894395ac0d65c2f36b10f8ad080ca7a14b999c5022e1dd8c9066f74fa2be6"
Jul 29 17:05:23.430984 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.430964    2570 scope.go:117] "RemoveContainer" containerID="d8a7d741d31bff7f8e302cac7883b80727245e90865358a06d8fd47fa22d6188"
Jul 29 17:05:23.435582 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.435556    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"]
Jul 29 17:05:23.440049 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.440028    2570 scope.go:117] "RemoveContainer" containerID="1d1d92bbf5907526934301cdc462fc131649eab05e866ebdd4a577eba7c9959e"
Jul 29 17:05:23.440354 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:05:23.440330    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1d92bbf5907526934301cdc462fc131649eab05e866ebdd4a577eba7c9959e\": container with ID starting with 1d1d92bbf5907526934301cdc462fc131649eab05e866ebdd4a577eba7c9959e not found: ID does not exist" containerID="1d1d92bbf5907526934301cdc462fc131649eab05e866ebdd4a577eba7c9959e"
Jul 29 17:05:23.440453 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.440362    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1d92bbf5907526934301cdc462fc131649eab05e866ebdd4a577eba7c9959e"} err="failed to get container status \"1d1d92bbf5907526934301cdc462fc131649eab05e866ebdd4a577eba7c9959e\": rpc error: code = NotFound desc = could not find container \"1d1d92bbf5907526934301cdc462fc131649eab05e866ebdd4a577eba7c9959e\": container with ID starting with 1d1d92bbf5907526934301cdc462fc131649eab05e866ebdd4a577eba7c9959e not found: ID does not exist"
Jul 29 17:05:23.440453 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.440381    2570 scope.go:117] "RemoveContainer" containerID="e42894395ac0d65c2f36b10f8ad080ca7a14b999c5022e1dd8c9066f74fa2be6"
Jul 29 17:05:23.440662 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:05:23.440644    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42894395ac0d65c2f36b10f8ad080ca7a14b999c5022e1dd8c9066f74fa2be6\": container with ID starting with e42894395ac0d65c2f36b10f8ad080ca7a14b999c5022e1dd8c9066f74fa2be6 not found: ID does not exist" containerID="e42894395ac0d65c2f36b10f8ad080ca7a14b999c5022e1dd8c9066f74fa2be6"
Jul 29 17:05:23.440708 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.440668    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42894395ac0d65c2f36b10f8ad080ca7a14b999c5022e1dd8c9066f74fa2be6"} err="failed to get container status \"e42894395ac0d65c2f36b10f8ad080ca7a14b999c5022e1dd8c9066f74fa2be6\": rpc error: code = NotFound desc = could not find container \"e42894395ac0d65c2f36b10f8ad080ca7a14b999c5022e1dd8c9066f74fa2be6\": container with ID starting with e42894395ac0d65c2f36b10f8ad080ca7a14b999c5022e1dd8c9066f74fa2be6 not found: ID does not exist"
Jul 29 17:05:23.440708 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.440684    2570 scope.go:117] "RemoveContainer" containerID="d8a7d741d31bff7f8e302cac7883b80727245e90865358a06d8fd47fa22d6188"
Jul 29 17:05:23.440923 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:05:23.440906    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a7d741d31bff7f8e302cac7883b80727245e90865358a06d8fd47fa22d6188\": container with ID starting with d8a7d741d31bff7f8e302cac7883b80727245e90865358a06d8fd47fa22d6188 not found: ID does not exist" containerID="d8a7d741d31bff7f8e302cac7883b80727245e90865358a06d8fd47fa22d6188"
Jul 29 17:05:23.440967 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.440929    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a7d741d31bff7f8e302cac7883b80727245e90865358a06d8fd47fa22d6188"} err="failed to get container status \"d8a7d741d31bff7f8e302cac7883b80727245e90865358a06d8fd47fa22d6188\": rpc error: code = NotFound desc = could not find container \"d8a7d741d31bff7f8e302cac7883b80727245e90865358a06d8fd47fa22d6188\": container with ID starting with d8a7d741d31bff7f8e302cac7883b80727245e90865358a06d8fd47fa22d6188 not found: ID does not exist"
Jul 29 17:05:23.441831 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.441813    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5f62m8h"]
Jul 29 17:05:23.914872 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:23.914834    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="137d69a8-afc1-4dfb-8de5-7e120a29f16f" path="/var/lib/kubelet/pods/137d69a8-afc1-4dfb-8de5-7e120a29f16f/volumes"
Jul 29 17:05:24.473891 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.473844    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"]
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474270    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f5cdee-4903-4422-8e02-d02ac106f01c" containerName="tokenizer"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474282    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f5cdee-4903-4422-8e02-d02ac106f01c" containerName="tokenizer"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474294    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f5cdee-4903-4422-8e02-d02ac106f01c" containerName="main"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474300    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f5cdee-4903-4422-8e02-d02ac106f01c" containerName="main"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474306    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f5cdee-4903-4422-8e02-d02ac106f01c" containerName="storage-initializer"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474313    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f5cdee-4903-4422-8e02-d02ac106f01c" containerName="storage-initializer"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474320    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137d69a8-afc1-4dfb-8de5-7e120a29f16f" containerName="main"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474328    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="137d69a8-afc1-4dfb-8de5-7e120a29f16f" containerName="main"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474346    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de4ac348-05fb-41dd-ae2c-437bf8f262aa" containerName="storage-initializer"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474351    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="de4ac348-05fb-41dd-ae2c-437bf8f262aa" containerName="storage-initializer"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474361    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137d69a8-afc1-4dfb-8de5-7e120a29f16f" containerName="storage-initializer"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474366    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="137d69a8-afc1-4dfb-8de5-7e120a29f16f" containerName="storage-initializer"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474372    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137d69a8-afc1-4dfb-8de5-7e120a29f16f" containerName="tokenizer"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474376    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="137d69a8-afc1-4dfb-8de5-7e120a29f16f" containerName="tokenizer"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474383    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de4ac348-05fb-41dd-ae2c-437bf8f262aa" containerName="main"
Jul 29 17:05:24.474413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474388    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="de4ac348-05fb-41dd-ae2c-437bf8f262aa" containerName="main"
Jul 29 17:05:24.475296 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474460    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="137d69a8-afc1-4dfb-8de5-7e120a29f16f" containerName="tokenizer"
Jul 29 17:05:24.475296 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474469    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="137d69a8-afc1-4dfb-8de5-7e120a29f16f" containerName="main"
Jul 29 17:05:24.475296 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474478    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f5cdee-4903-4422-8e02-d02ac106f01c" containerName="main"
Jul 29 17:05:24.475296 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474488    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="de4ac348-05fb-41dd-ae2c-437bf8f262aa" containerName="main"
Jul 29 17:05:24.475296 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.474495    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f5cdee-4903-4422-8e02-d02ac106f01c" containerName="tokenizer"
Jul 29 17:05:24.479177 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.479142    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.482434 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.482407    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-szzmj\""
Jul 29 17:05:24.482434 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.482432    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 17:05:24.482635 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.482411    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 29 17:05:24.482635 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.482407    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 17:05:24.488189 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.488145    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"]
Jul 29 17:05:24.492852 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.492831    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-home\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.492977 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.492862    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-tmp-dir\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.492977 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.492894    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-model-cache\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.492977 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.492943    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa51433-9bac-4542-a98d-20651c1aa64d-tls-certs\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.493087 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.492988    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.493087 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.493052    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-dshm\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.493151 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.493089    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpbmn\" (UniqueName: \"kubernetes.io/projected/4aa51433-9bac-4542-a98d-20651c1aa64d-kube-api-access-jpbmn\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.593904 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.593859    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.593904 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.593910    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-dshm\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.594198 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.593931    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpbmn\" (UniqueName: \"kubernetes.io/projected/4aa51433-9bac-4542-a98d-20651c1aa64d-kube-api-access-jpbmn\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.594198 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.593983    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-home\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.594198 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.594003    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-tmp-dir\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.594198 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.594021    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-model-cache\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.594198 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.594041    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa51433-9bac-4542-a98d-20651c1aa64d-tls-certs\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.594478 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.594361    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.594478 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.594431    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-home\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.594551 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.594526    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-tmp-dir\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.594589 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.594543    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-model-cache\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.596318 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.596296    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-dshm\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.596573 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.596556    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa51433-9bac-4542-a98d-20651c1aa64d-tls-certs\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.602335 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.602313    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpbmn\" (UniqueName: \"kubernetes.io/projected/4aa51433-9bac-4542-a98d-20651c1aa64d-kube-api-access-jpbmn\") pod \"precise-prefix-cache-test-kserve-66f68489b9-bq54t\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.695071 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.695029    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"]
Jul 29 17:05:24.700713 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.700685    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.702944 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.702922    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-lxswj\""
Jul 29 17:05:24.709297 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.709272    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"]
Jul 29 17:05:24.791089 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.791053    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:24.795302 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.795278    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.795385 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.795341    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.795424 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.795390    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xzqv\" (UniqueName: \"kubernetes.io/projected/3508d12d-ca1e-4b98-9509-c805f8b5a53a-kube-api-access-6xzqv\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.795424 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.795411    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.795491 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.795437    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.795491 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.795462    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.896979 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.896929    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.897209 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.897021    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.897209 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.897070    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xzqv\" (UniqueName: \"kubernetes.io/projected/3508d12d-ca1e-4b98-9509-c805f8b5a53a-kube-api-access-6xzqv\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.897209 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.897103    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.897209 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.897153    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.897510 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.897230    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.897510 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.897407    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.897615 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.897569    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.897701 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.897655    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.897784 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.897710    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.900215 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.900189    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.905956 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.905924    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xzqv\" (UniqueName: \"kubernetes.io/projected/3508d12d-ca1e-4b98-9509-c805f8b5a53a-kube-api-access-6xzqv\") pod \"precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:24.933736 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:24.933711    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"]
Jul 29 17:05:24.935938 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:05:24.935909    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa51433_9bac_4542_a98d_20651c1aa64d.slice/crio-e8d9822a35a243ace57ea21bc500ffa55479f0996f4df023847e4e2cfc67bbe6 WatchSource:0}: Error finding container e8d9822a35a243ace57ea21bc500ffa55479f0996f4df023847e4e2cfc67bbe6: Status 404 returned error can't find the container with id e8d9822a35a243ace57ea21bc500ffa55479f0996f4df023847e4e2cfc67bbe6
Jul 29 17:05:25.013637 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:25.013602    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:25.164122 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:25.164098    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"]
Jul 29 17:05:25.165811 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:05:25.165782    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3508d12d_ca1e_4b98_9509_c805f8b5a53a.slice/crio-863cf876e6344abaf1569dc95ef723531d4f742c6ecd538b730254d03b003040 WatchSource:0}: Error finding container 863cf876e6344abaf1569dc95ef723531d4f742c6ecd538b730254d03b003040: Status 404 returned error can't find the container with id 863cf876e6344abaf1569dc95ef723531d4f742c6ecd538b730254d03b003040
Jul 29 17:05:25.434530 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:25.434436    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md" event={"ID":"3508d12d-ca1e-4b98-9509-c805f8b5a53a","Type":"ContainerStarted","Data":"a8fe2a9419652807ce0d51ef25abdcc548bfb9618d1b44128d9cd661ffbc8016"}
Jul 29 17:05:25.434530 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:25.434502    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md" event={"ID":"3508d12d-ca1e-4b98-9509-c805f8b5a53a","Type":"ContainerStarted","Data":"863cf876e6344abaf1569dc95ef723531d4f742c6ecd538b730254d03b003040"}
Jul 29 17:05:25.436105 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:25.436081    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t" event={"ID":"4aa51433-9bac-4542-a98d-20651c1aa64d","Type":"ContainerStarted","Data":"5c349d9321a0c0e2cb3428d19acd51f95fdcad972faa65bca2828d84f59eda92"}
Jul 29 17:05:25.436235 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:25.436127    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t" event={"ID":"4aa51433-9bac-4542-a98d-20651c1aa64d","Type":"ContainerStarted","Data":"e8d9822a35a243ace57ea21bc500ffa55479f0996f4df023847e4e2cfc67bbe6"}
Jul 29 17:05:26.444900 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:26.444863    2570 generic.go:356] "Generic (PLEG): container finished" podID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerID="a8fe2a9419652807ce0d51ef25abdcc548bfb9618d1b44128d9cd661ffbc8016" exitCode=0
Jul 29 17:05:26.445487 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:26.444945    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md" event={"ID":"3508d12d-ca1e-4b98-9509-c805f8b5a53a","Type":"ContainerDied","Data":"a8fe2a9419652807ce0d51ef25abdcc548bfb9618d1b44128d9cd661ffbc8016"}
Jul 29 17:05:27.455666 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:27.455623    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md" event={"ID":"3508d12d-ca1e-4b98-9509-c805f8b5a53a","Type":"ContainerStarted","Data":"fae260b58781526d3f231d7b4b1980d0223b2fbd9dbcc10438da9b8d309ed5cf"}
Jul 29 17:05:27.456061 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:27.455688    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md" event={"ID":"3508d12d-ca1e-4b98-9509-c805f8b5a53a","Type":"ContainerStarted","Data":"ac6b84d132bc96065cb497976f453a914c67b086877a41074fa9f8537e6283ad"}
Jul 29 17:05:27.476798 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:27.476741    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md" podStartSLOduration=2.753084491 podStartE2EDuration="3.476725502s" totalImagesPullingTime="0s" totalInitContainerRuntime="723.641011ms" isStatefulPod=false podCreationTimestamp="2026-07-29 17:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:05:27.475827943 +0000 UTC m=+855.965347601" watchObservedRunningTime="2026-07-29 17:05:27.476725502 +0000 UTC m=+855.966245157"
Jul 29 17:05:28.461939 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:28.461907    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:29.468369 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:29.468338    2570 generic.go:356] "Generic (PLEG): container finished" podID="4aa51433-9bac-4542-a98d-20651c1aa64d" containerID="5c349d9321a0c0e2cb3428d19acd51f95fdcad972faa65bca2828d84f59eda92" exitCode=0
Jul 29 17:05:29.468776 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:29.468429    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t" event={"ID":"4aa51433-9bac-4542-a98d-20651c1aa64d","Type":"ContainerDied","Data":"5c349d9321a0c0e2cb3428d19acd51f95fdcad972faa65bca2828d84f59eda92"}
Jul 29 17:05:30.480594 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:30.480552    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t" event={"ID":"4aa51433-9bac-4542-a98d-20651c1aa64d","Type":"ContainerStarted","Data":"2858866601f2c07ec548b467a69b50fc1a9111fbebe86d4f2add28f60acb065d"}
Jul 29 17:05:30.503810 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:30.503739    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t" podStartSLOduration=2.186399255 podStartE2EDuration="6.503714456s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.317315201s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:05:30.501785025 +0000 UTC m=+858.991304682" watchObservedRunningTime="2026-07-29 17:05:30.503714456 +0000 UTC m=+858.993234168"
Jul 29 17:05:34.791377 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:34.791319    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:34.791789 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:34.791450    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:34.804350 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:34.804320    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:35.014851 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:35.014820    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:35.015048 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:35.014883    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:35.016210 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:05:35.016183    2570 logging.go:55] [core] [Channel #83 SubChannel #84]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.53:9003", ServerName: "10.132.0.53:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.53:9003: connect: connection refused"
Jul 29 17:05:35.017663 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:35.017640    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:35.510459 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:35.510429    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:35.520753 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:35.520728    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:36.015190 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:36.015121    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md" podUID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.53:9003\" within 1s: context deadline exceeded"
Jul 29 17:05:45.014294 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:05:45.014259    2570 logging.go:55] [core] [Channel #85 SubChannel #86]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.53:9003", ServerName: "10.132.0.53:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.53:9003: connect: connection refused"
Jul 29 17:05:46.014153 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:46.014105    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md" podUID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.53:9003\" within 1s: context deadline exceeded"
Jul 29 17:05:56.516426 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:56.516397    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:57.966618 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:57.966567    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"]
Jul 29 17:05:57.967497 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:57.967435    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t" podUID="4aa51433-9bac-4542-a98d-20651c1aa64d" containerName="main" containerID="cri-o://2858866601f2c07ec548b467a69b50fc1a9111fbebe86d4f2add28f60acb065d" gracePeriod=30
Jul 29 17:05:57.974580 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:57.974551    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"]
Jul 29 17:05:57.975264 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:57.974799    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md" podUID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerName="main" containerID="cri-o://ac6b84d132bc96065cb497976f453a914c67b086877a41074fa9f8537e6283ad" gracePeriod=30
Jul 29 17:05:57.975264 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:57.974839    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md" podUID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerName="tokenizer" containerID="cri-o://fae260b58781526d3f231d7b4b1980d0223b2fbd9dbcc10438da9b8d309ed5cf" gracePeriod=30
Jul 29 17:05:58.237469 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.237440    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:58.412008 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.411975    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa51433-9bac-4542-a98d-20651c1aa64d-tls-certs\") pod \"4aa51433-9bac-4542-a98d-20651c1aa64d\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") "
Jul 29 17:05:58.412207 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.412044    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-model-cache\") pod \"4aa51433-9bac-4542-a98d-20651c1aa64d\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") "
Jul 29 17:05:58.412207 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.412077    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-tmp-dir\") pod \"4aa51433-9bac-4542-a98d-20651c1aa64d\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") "
Jul 29 17:05:58.412207 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.412151    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-dshm\") pod \"4aa51433-9bac-4542-a98d-20651c1aa64d\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") "
Jul 29 17:05:58.412207 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.412201    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-home\") pod \"4aa51433-9bac-4542-a98d-20651c1aa64d\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") "
Jul 29 17:05:58.412434 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.412231    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-kserve-provision-location\") pod \"4aa51433-9bac-4542-a98d-20651c1aa64d\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") "
Jul 29 17:05:58.412434 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.412258    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpbmn\" (UniqueName: \"kubernetes.io/projected/4aa51433-9bac-4542-a98d-20651c1aa64d-kube-api-access-jpbmn\") pod \"4aa51433-9bac-4542-a98d-20651c1aa64d\" (UID: \"4aa51433-9bac-4542-a98d-20651c1aa64d\") "
Jul 29 17:05:58.412434 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.412390    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-model-cache" (OuterVolumeSpecName: "model-cache") pod "4aa51433-9bac-4542-a98d-20651c1aa64d" (UID: "4aa51433-9bac-4542-a98d-20651c1aa64d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:58.412434 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.412401    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4aa51433-9bac-4542-a98d-20651c1aa64d" (UID: "4aa51433-9bac-4542-a98d-20651c1aa64d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:58.412434 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.412416    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-home" (OuterVolumeSpecName: "home") pod "4aa51433-9bac-4542-a98d-20651c1aa64d" (UID: "4aa51433-9bac-4542-a98d-20651c1aa64d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:58.412697 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.412538    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:58.412697 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.412557    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:58.412697 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.412573    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:58.414922 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.414892    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-dshm" (OuterVolumeSpecName: "dshm") pod "4aa51433-9bac-4542-a98d-20651c1aa64d" (UID: "4aa51433-9bac-4542-a98d-20651c1aa64d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:58.414922 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.414901    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa51433-9bac-4542-a98d-20651c1aa64d-kube-api-access-jpbmn" (OuterVolumeSpecName: "kube-api-access-jpbmn") pod "4aa51433-9bac-4542-a98d-20651c1aa64d" (UID: "4aa51433-9bac-4542-a98d-20651c1aa64d"). InnerVolumeSpecName "kube-api-access-jpbmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:05:58.415128 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.414934    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa51433-9bac-4542-a98d-20651c1aa64d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4aa51433-9bac-4542-a98d-20651c1aa64d" (UID: "4aa51433-9bac-4542-a98d-20651c1aa64d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:05:58.470465 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.470415    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4aa51433-9bac-4542-a98d-20651c1aa64d" (UID: "4aa51433-9bac-4542-a98d-20651c1aa64d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:58.513401 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.513361    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa51433-9bac-4542-a98d-20651c1aa64d-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:58.513401 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.513396    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:58.513401 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.513410    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aa51433-9bac-4542-a98d-20651c1aa64d-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:58.513646 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.513423    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpbmn\" (UniqueName: \"kubernetes.io/projected/4aa51433-9bac-4542-a98d-20651c1aa64d-kube-api-access-jpbmn\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:58.638386 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.638350    2570 generic.go:356] "Generic (PLEG): container finished" podID="4aa51433-9bac-4542-a98d-20651c1aa64d" containerID="2858866601f2c07ec548b467a69b50fc1a9111fbebe86d4f2add28f60acb065d" exitCode=0
Jul 29 17:05:58.638568 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.638399    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"
Jul 29 17:05:58.638568 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.638399    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t" event={"ID":"4aa51433-9bac-4542-a98d-20651c1aa64d","Type":"ContainerDied","Data":"2858866601f2c07ec548b467a69b50fc1a9111fbebe86d4f2add28f60acb065d"}
Jul 29 17:05:58.638568 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.638532    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t" event={"ID":"4aa51433-9bac-4542-a98d-20651c1aa64d","Type":"ContainerDied","Data":"e8d9822a35a243ace57ea21bc500ffa55479f0996f4df023847e4e2cfc67bbe6"}
Jul 29 17:05:58.638568 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.638567    2570 scope.go:117] "RemoveContainer" containerID="2858866601f2c07ec548b467a69b50fc1a9111fbebe86d4f2add28f60acb065d"
Jul 29 17:05:58.641040 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.640986    2570 generic.go:356] "Generic (PLEG): container finished" podID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerID="ac6b84d132bc96065cb497976f453a914c67b086877a41074fa9f8537e6283ad" exitCode=0
Jul 29 17:05:58.641040 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.641024    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md" event={"ID":"3508d12d-ca1e-4b98-9509-c805f8b5a53a","Type":"ContainerDied","Data":"ac6b84d132bc96065cb497976f453a914c67b086877a41074fa9f8537e6283ad"}
Jul 29 17:05:58.649543 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.649524    2570 scope.go:117] "RemoveContainer" containerID="5c349d9321a0c0e2cb3428d19acd51f95fdcad972faa65bca2828d84f59eda92"
Jul 29 17:05:58.662545 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.662511    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"]
Jul 29 17:05:58.668921 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.668894    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-bq54t"]
Jul 29 17:05:58.671858 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.671831    2570 scope.go:117] "RemoveContainer" containerID="2858866601f2c07ec548b467a69b50fc1a9111fbebe86d4f2add28f60acb065d"
Jul 29 17:05:58.672311 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:05:58.672282    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2858866601f2c07ec548b467a69b50fc1a9111fbebe86d4f2add28f60acb065d\": container with ID starting with 2858866601f2c07ec548b467a69b50fc1a9111fbebe86d4f2add28f60acb065d not found: ID does not exist" containerID="2858866601f2c07ec548b467a69b50fc1a9111fbebe86d4f2add28f60acb065d"
Jul 29 17:05:58.672435 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.672323    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2858866601f2c07ec548b467a69b50fc1a9111fbebe86d4f2add28f60acb065d"} err="failed to get container status \"2858866601f2c07ec548b467a69b50fc1a9111fbebe86d4f2add28f60acb065d\": rpc error: code = NotFound desc = could not find container \"2858866601f2c07ec548b467a69b50fc1a9111fbebe86d4f2add28f60acb065d\": container with ID starting with 2858866601f2c07ec548b467a69b50fc1a9111fbebe86d4f2add28f60acb065d not found: ID does not exist"
Jul 29 17:05:58.672435 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.672351    2570 scope.go:117] "RemoveContainer" containerID="5c349d9321a0c0e2cb3428d19acd51f95fdcad972faa65bca2828d84f59eda92"
Jul 29 17:05:58.672731 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:05:58.672703    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c349d9321a0c0e2cb3428d19acd51f95fdcad972faa65bca2828d84f59eda92\": container with ID starting with 5c349d9321a0c0e2cb3428d19acd51f95fdcad972faa65bca2828d84f59eda92 not found: ID does not exist" containerID="5c349d9321a0c0e2cb3428d19acd51f95fdcad972faa65bca2828d84f59eda92"
Jul 29 17:05:58.672831 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:58.672739    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c349d9321a0c0e2cb3428d19acd51f95fdcad972faa65bca2828d84f59eda92"} err="failed to get container status \"5c349d9321a0c0e2cb3428d19acd51f95fdcad972faa65bca2828d84f59eda92\": rpc error: code = NotFound desc = could not find container \"5c349d9321a0c0e2cb3428d19acd51f95fdcad972faa65bca2828d84f59eda92\": container with ID starting with 5c349d9321a0c0e2cb3428d19acd51f95fdcad972faa65bca2828d84f59eda92 not found: ID does not exist"
Jul 29 17:05:59.635551 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.635524    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:59.648620 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.648534    2570 generic.go:356] "Generic (PLEG): container finished" podID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerID="fae260b58781526d3f231d7b4b1980d0223b2fbd9dbcc10438da9b8d309ed5cf" exitCode=0
Jul 29 17:05:59.648620 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.648587    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md" event={"ID":"3508d12d-ca1e-4b98-9509-c805f8b5a53a","Type":"ContainerDied","Data":"fae260b58781526d3f231d7b4b1980d0223b2fbd9dbcc10438da9b8d309ed5cf"}
Jul 29 17:05:59.648620 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.648608    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"
Jul 29 17:05:59.648856 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.648655    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md" event={"ID":"3508d12d-ca1e-4b98-9509-c805f8b5a53a","Type":"ContainerDied","Data":"863cf876e6344abaf1569dc95ef723531d4f742c6ecd538b730254d03b003040"}
Jul 29 17:05:59.648856 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.648678    2570 scope.go:117] "RemoveContainer" containerID="fae260b58781526d3f231d7b4b1980d0223b2fbd9dbcc10438da9b8d309ed5cf"
Jul 29 17:05:59.658436 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.658416    2570 scope.go:117] "RemoveContainer" containerID="ac6b84d132bc96065cb497976f453a914c67b086877a41074fa9f8537e6283ad"
Jul 29 17:05:59.669236 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.669215    2570 scope.go:117] "RemoveContainer" containerID="a8fe2a9419652807ce0d51ef25abdcc548bfb9618d1b44128d9cd661ffbc8016"
Jul 29 17:05:59.677217 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.677192    2570 scope.go:117] "RemoveContainer" containerID="fae260b58781526d3f231d7b4b1980d0223b2fbd9dbcc10438da9b8d309ed5cf"
Jul 29 17:05:59.677513 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:05:59.677493    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae260b58781526d3f231d7b4b1980d0223b2fbd9dbcc10438da9b8d309ed5cf\": container with ID starting with fae260b58781526d3f231d7b4b1980d0223b2fbd9dbcc10438da9b8d309ed5cf not found: ID does not exist" containerID="fae260b58781526d3f231d7b4b1980d0223b2fbd9dbcc10438da9b8d309ed5cf"
Jul 29 17:05:59.677569 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.677523    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae260b58781526d3f231d7b4b1980d0223b2fbd9dbcc10438da9b8d309ed5cf"} err="failed to get container status \"fae260b58781526d3f231d7b4b1980d0223b2fbd9dbcc10438da9b8d309ed5cf\": rpc error: code = NotFound desc = could not find container \"fae260b58781526d3f231d7b4b1980d0223b2fbd9dbcc10438da9b8d309ed5cf\": container with ID starting with fae260b58781526d3f231d7b4b1980d0223b2fbd9dbcc10438da9b8d309ed5cf not found: ID does not exist"
Jul 29 17:05:59.677569 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.677547    2570 scope.go:117] "RemoveContainer" containerID="ac6b84d132bc96065cb497976f453a914c67b086877a41074fa9f8537e6283ad"
Jul 29 17:05:59.677857 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:05:59.677814    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6b84d132bc96065cb497976f453a914c67b086877a41074fa9f8537e6283ad\": container with ID starting with ac6b84d132bc96065cb497976f453a914c67b086877a41074fa9f8537e6283ad not found: ID does not exist" containerID="ac6b84d132bc96065cb497976f453a914c67b086877a41074fa9f8537e6283ad"
Jul 29 17:05:59.677942 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.677864    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6b84d132bc96065cb497976f453a914c67b086877a41074fa9f8537e6283ad"} err="failed to get container status \"ac6b84d132bc96065cb497976f453a914c67b086877a41074fa9f8537e6283ad\": rpc error: code = NotFound desc = could not find container \"ac6b84d132bc96065cb497976f453a914c67b086877a41074fa9f8537e6283ad\": container with ID starting with ac6b84d132bc96065cb497976f453a914c67b086877a41074fa9f8537e6283ad not found: ID does not exist"
Jul 29 17:05:59.677942 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.677884    2570 scope.go:117] "RemoveContainer" containerID="a8fe2a9419652807ce0d51ef25abdcc548bfb9618d1b44128d9cd661ffbc8016"
Jul 29 17:05:59.678232 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:05:59.678211    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8fe2a9419652807ce0d51ef25abdcc548bfb9618d1b44128d9cd661ffbc8016\": container with ID starting with a8fe2a9419652807ce0d51ef25abdcc548bfb9618d1b44128d9cd661ffbc8016 not found: ID does not exist" containerID="a8fe2a9419652807ce0d51ef25abdcc548bfb9618d1b44128d9cd661ffbc8016"
Jul 29 17:05:59.678299 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.678237    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8fe2a9419652807ce0d51ef25abdcc548bfb9618d1b44128d9cd661ffbc8016"} err="failed to get container status \"a8fe2a9419652807ce0d51ef25abdcc548bfb9618d1b44128d9cd661ffbc8016\": rpc error: code = NotFound desc = could not find container \"a8fe2a9419652807ce0d51ef25abdcc548bfb9618d1b44128d9cd661ffbc8016\": container with ID starting with a8fe2a9419652807ce0d51ef25abdcc548bfb9618d1b44128d9cd661ffbc8016 not found: ID does not exist"
Jul 29 17:05:59.724667 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.724637    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tls-certs\") pod \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") "
Jul 29 17:05:59.724667 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.724681    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xzqv\" (UniqueName: \"kubernetes.io/projected/3508d12d-ca1e-4b98-9509-c805f8b5a53a-kube-api-access-6xzqv\") pod \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") "
Jul 29 17:05:59.724890 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.724701    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-kserve-provision-location\") pod \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") "
Jul 29 17:05:59.724890 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.724740    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-tmp\") pod \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") "
Jul 29 17:05:59.724890 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.724773    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-cache\") pod \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") "
Jul 29 17:05:59.724890 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.724837    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-uds\") pod \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\" (UID: \"3508d12d-ca1e-4b98-9509-c805f8b5a53a\") "
Jul 29 17:05:59.725217 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.725121    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "3508d12d-ca1e-4b98-9509-c805f8b5a53a" (UID: "3508d12d-ca1e-4b98-9509-c805f8b5a53a"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:59.725310 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.725221    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3508d12d-ca1e-4b98-9509-c805f8b5a53a" (UID: "3508d12d-ca1e-4b98-9509-c805f8b5a53a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:59.725310 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.725272    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "3508d12d-ca1e-4b98-9509-c805f8b5a53a" (UID: "3508d12d-ca1e-4b98-9509-c805f8b5a53a"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:59.725516 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.725490    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3508d12d-ca1e-4b98-9509-c805f8b5a53a" (UID: "3508d12d-ca1e-4b98-9509-c805f8b5a53a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:59.726831 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.726808    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3508d12d-ca1e-4b98-9509-c805f8b5a53a-kube-api-access-6xzqv" (OuterVolumeSpecName: "kube-api-access-6xzqv") pod "3508d12d-ca1e-4b98-9509-c805f8b5a53a" (UID: "3508d12d-ca1e-4b98-9509-c805f8b5a53a"). InnerVolumeSpecName "kube-api-access-6xzqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:05:59.726924 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.726900    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3508d12d-ca1e-4b98-9509-c805f8b5a53a" (UID: "3508d12d-ca1e-4b98-9509-c805f8b5a53a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:05:59.826486 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.826449    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-tmp\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:59.826486 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.826477    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:59.826486 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.826488    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tokenizer-uds\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:59.826486 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.826499    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3508d12d-ca1e-4b98-9509-c805f8b5a53a-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:59.826755 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.826507    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xzqv\" (UniqueName: \"kubernetes.io/projected/3508d12d-ca1e-4b98-9509-c805f8b5a53a-kube-api-access-6xzqv\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:59.826755 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.826517    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3508d12d-ca1e-4b98-9509-c805f8b5a53a-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:05:59.914082 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.914001    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aa51433-9bac-4542-a98d-20651c1aa64d" path="/var/lib/kubelet/pods/4aa51433-9bac-4542-a98d-20651c1aa64d/volumes"
Jul 29 17:05:59.966278 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.966240    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"]
Jul 29 17:05:59.969725 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:05:59.969698    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-79b674bd2h5md"]
Jul 29 17:06:01.914777 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:01.914741    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" path="/var/lib/kubelet/pods/3508d12d-ca1e-4b98-9509-c805f8b5a53a/volumes"
Jul 29 17:06:11.883473 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:11.883435    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:06:11.885640 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:11.885614    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:06:12.077975 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.077936    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"]
Jul 29 17:06:12.078398 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.078380    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerName="main"
Jul 29 17:06:12.078398 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.078399    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerName="main"
Jul 29 17:06:12.078548 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.078422    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerName="tokenizer"
Jul 29 17:06:12.078548 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.078431    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerName="tokenizer"
Jul 29 17:06:12.078548 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.078439    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa51433-9bac-4542-a98d-20651c1aa64d" containerName="main"
Jul 29 17:06:12.078548 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.078447    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa51433-9bac-4542-a98d-20651c1aa64d" containerName="main"
Jul 29 17:06:12.078548 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.078456    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa51433-9bac-4542-a98d-20651c1aa64d" containerName="storage-initializer"
Jul 29 17:06:12.078548 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.078464    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa51433-9bac-4542-a98d-20651c1aa64d" containerName="storage-initializer"
Jul 29 17:06:12.078548 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.078474    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerName="storage-initializer"
Jul 29 17:06:12.078548 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.078482    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerName="storage-initializer"
Jul 29 17:06:12.078937 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.078583    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerName="tokenizer"
Jul 29 17:06:12.078937 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.078599    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aa51433-9bac-4542-a98d-20651c1aa64d" containerName="main"
Jul 29 17:06:12.078937 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.078610    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3508d12d-ca1e-4b98-9509-c805f8b5a53a" containerName="main"
Jul 29 17:06:12.083287 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.083260    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.085697 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.085669    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-szzmj\""
Jul 29 17:06:12.085833 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.085669    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 17:06:12.085833 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.085676    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 29 17:06:12.085833 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.085718    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 17:06:12.094636 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.094611    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"]
Jul 29 17:06:12.123484 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.123456    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-model-cache\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.123484 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.123496    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-home\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.123820 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.123578    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c695af8-a557-4416-a1f2-99da117161b5-tls-certs\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.123820 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.123609    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-tmp-dir\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.123820 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.123661    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-kserve-provision-location\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.123820 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.123710    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-dshm\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.123820 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.123754    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfhqv\" (UniqueName: \"kubernetes.io/projected/7c695af8-a557-4416-a1f2-99da117161b5-kube-api-access-cfhqv\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.224513 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.224423    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-model-cache\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.224513 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.224476    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-home\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.224513 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.224507    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c695af8-a557-4416-a1f2-99da117161b5-tls-certs\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.224823 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.224526    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-tmp-dir\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.224823 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.224646    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-kserve-provision-location\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.224823 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.224703    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-dshm\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.224823 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.224762    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfhqv\" (UniqueName: \"kubernetes.io/projected/7c695af8-a557-4416-a1f2-99da117161b5-kube-api-access-cfhqv\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.224823 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.224814    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-model-cache\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.225031 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.224874    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-home\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.225031 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.224921    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-tmp-dir\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.225109 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.225037    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-kserve-provision-location\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.226969 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.226950    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-dshm\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.227054 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.227039    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c695af8-a557-4416-a1f2-99da117161b5-tls-certs\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.233973 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.233940    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfhqv\" (UniqueName: \"kubernetes.io/projected/7c695af8-a557-4416-a1f2-99da117161b5-kube-api-access-cfhqv\") pod \"stop-feature-test-kserve-5697d48bb4-zdsjd\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.284577 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.284540    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"]
Jul 29 17:06:12.288788 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.288764    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.290949 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.290919    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-fq8k8\""
Jul 29 17:06:12.299376 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.299333    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"]
Jul 29 17:06:12.325812 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.325779    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.325812 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.325821    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.326068 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.325852    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.326068 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.325874    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.326068 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.325895    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krfch\" (UniqueName: \"kubernetes.io/projected/f8a77801-f479-434c-874f-4affb131b0c8-kube-api-access-krfch\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.326068 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.325973    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a77801-f479-434c-874f-4affb131b0c8-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.394301 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.394265    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:06:12.426694 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.426652    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a77801-f479-434c-874f-4affb131b0c8-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.426874 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.426800    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.426874 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.426846    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.426968 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.426892    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.426968 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.426923    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.427062 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.426967    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krfch\" (UniqueName: \"kubernetes.io/projected/f8a77801-f479-434c-874f-4affb131b0c8-kube-api-access-krfch\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.427249 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.427224    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.427249 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.427239    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.427480 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.427461    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.427480 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.427472    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.429256 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.429235    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a77801-f479-434c-874f-4affb131b0c8-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.435136 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.435103    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krfch\" (UniqueName: \"kubernetes.io/projected/f8a77801-f479-434c-874f-4affb131b0c8-kube-api-access-krfch\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-r2629\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.540337 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.540307    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"]
Jul 29 17:06:12.542008 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:06:12.541983    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c695af8_a557_4416_a1f2_99da117161b5.slice/crio-3000e9dc1531f701398659114aa6c3f821c06ce7b6f46ca25ad2247c8332f3ed WatchSource:0}: Error finding container 3000e9dc1531f701398659114aa6c3f821c06ce7b6f46ca25ad2247c8332f3ed: Status 404 returned error can't find the container with id 3000e9dc1531f701398659114aa6c3f821c06ce7b6f46ca25ad2247c8332f3ed
Jul 29 17:06:12.600282 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.600245    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:12.723480 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.723422    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" event={"ID":"7c695af8-a557-4416-a1f2-99da117161b5","Type":"ContainerStarted","Data":"f1d8569ad2626e991f85651368e4d131bd110eea12172e8be18a02bb9dd95558"}
Jul 29 17:06:12.723662 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.723498    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" event={"ID":"7c695af8-a557-4416-a1f2-99da117161b5","Type":"ContainerStarted","Data":"3000e9dc1531f701398659114aa6c3f821c06ce7b6f46ca25ad2247c8332f3ed"}
Jul 29 17:06:12.758187 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:12.758136    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"]
Jul 29 17:06:12.760764 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:06:12.760723    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a77801_f479_434c_874f_4affb131b0c8.slice/crio-ea147500c6ddf441e5b373e0c7d6ab55b2a90ef3c1a45e61292207430a74d2e0 WatchSource:0}: Error finding container ea147500c6ddf441e5b373e0c7d6ab55b2a90ef3c1a45e61292207430a74d2e0: Status 404 returned error can't find the container with id ea147500c6ddf441e5b373e0c7d6ab55b2a90ef3c1a45e61292207430a74d2e0
Jul 29 17:06:13.730697 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:13.730665    2570 generic.go:356] "Generic (PLEG): container finished" podID="f8a77801-f479-434c-874f-4affb131b0c8" containerID="0a33e3fdd4b765f0da037564d3ebd1dd811dd38ac1021e73348efa3e00ee61b1" exitCode=0
Jul 29 17:06:13.731226 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:13.730752    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629" event={"ID":"f8a77801-f479-434c-874f-4affb131b0c8","Type":"ContainerDied","Data":"0a33e3fdd4b765f0da037564d3ebd1dd811dd38ac1021e73348efa3e00ee61b1"}
Jul 29 17:06:13.731226 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:13.730814    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629" event={"ID":"f8a77801-f479-434c-874f-4affb131b0c8","Type":"ContainerStarted","Data":"ea147500c6ddf441e5b373e0c7d6ab55b2a90ef3c1a45e61292207430a74d2e0"}
Jul 29 17:06:14.739919 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:14.739883    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629" event={"ID":"f8a77801-f479-434c-874f-4affb131b0c8","Type":"ContainerStarted","Data":"13bdecfcec015369838cbe4cda88ec600c9ebbf6054763efb34eb524d97976eb"}
Jul 29 17:06:14.740479 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:14.739943    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629" event={"ID":"f8a77801-f479-434c-874f-4affb131b0c8","Type":"ContainerStarted","Data":"56154d75c5d55919727a76ec53cb6f4cc2320fd2b118500f1f25016bd593aba2"}
Jul 29 17:06:14.740479 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:14.740046    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:14.771251 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:14.771182    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629" podStartSLOduration=2.042645996 podStartE2EDuration="2.77114659s" totalImagesPullingTime="0s" totalInitContainerRuntime="728.500594ms" isStatefulPod=false podCreationTimestamp="2026-07-29 17:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:06:14.764373105 +0000 UTC m=+903.253892762" watchObservedRunningTime="2026-07-29 17:06:14.77114659 +0000 UTC m=+903.260666247"
Jul 29 17:06:17.760776 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:17.760739    2570 generic.go:356] "Generic (PLEG): container finished" podID="7c695af8-a557-4416-a1f2-99da117161b5" containerID="f1d8569ad2626e991f85651368e4d131bd110eea12172e8be18a02bb9dd95558" exitCode=0
Jul 29 17:06:17.761227 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:17.760817    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" event={"ID":"7c695af8-a557-4416-a1f2-99da117161b5","Type":"ContainerDied","Data":"f1d8569ad2626e991f85651368e4d131bd110eea12172e8be18a02bb9dd95558"}
Jul 29 17:06:22.600875 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:22.600812    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:22.601408 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:22.600909    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:22.602434 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:22.602406    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629" podUID="f8a77801-f479-434c-874f-4affb131b0c8" containerName="tokenizer" probeResult="failure" output="Get \"http://10.132.0.55:8082/healthz\": dial tcp 10.132.0.55:8082: connect: connection refused"
Jul 29 17:06:32.602866 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:32.602585    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:32.623991 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:32.604095    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:06:52.864152 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:06:52.864117    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:07:10.083369 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:07:10.083329    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" event={"ID":"7c695af8-a557-4416-a1f2-99da117161b5","Type":"ContainerStarted","Data":"520dcb6e0f40c944f55c96d454ebe0eb01d4bb0899d14b0cf26188d98a611292"}
Jul 29 17:07:10.105742 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:07:10.105682    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" podStartSLOduration=1.552419278 podStartE2EDuration="58.105661583s" totalImagesPullingTime="51.559834837s" totalInitContainerRuntime="4.993407468s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:07:10.102667626 +0000 UTC m=+958.592187283" watchObservedRunningTime="2026-07-29 17:07:10.105661583 +0000 UTC m=+958.595181242"
Jul 29 17:07:12.394732 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:07:12.394688    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:07:12.395255 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:07:12.394766    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:07:12.396461 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:07:12.396429    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 29 17:07:22.395390 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:07:22.395345    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 29 17:07:32.395034 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:07:32.394990    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 29 17:07:42.395192 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:07:42.395067    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 29 17:07:52.395246 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:07:52.395196    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 29 17:08:02.395075 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:02.395022    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 29 17:08:04.377212 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.377177    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"]
Jul 29 17:08:04.385632 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.385600    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.388628 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.388597    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
Jul 29 17:08:04.391405 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.391383    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"]
Jul 29 17:08:04.462100 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.462054    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.462100 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.462101    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.462357 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.462124    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6809b2de-ff62-492a-92fb-52cf614b534f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.462408 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.462376    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjhvn\" (UniqueName: \"kubernetes.io/projected/6809b2de-ff62-492a-92fb-52cf614b534f-kube-api-access-bjhvn\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.462455 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.462431    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.462501 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.462459    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.462552 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.462510    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.563296 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.563259    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6809b2de-ff62-492a-92fb-52cf614b534f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.563476 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.563308    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjhvn\" (UniqueName: \"kubernetes.io/projected/6809b2de-ff62-492a-92fb-52cf614b534f-kube-api-access-bjhvn\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.563527 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.563491    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.563579 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.563527    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.563641 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.563580    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.563698 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.563658    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.563698 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.563693    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.564142 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.564109    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.564303 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.564122    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.564303 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.564204    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.564303 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.564208    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.565871 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.565843    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.565971 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.565858    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6809b2de-ff62-492a-92fb-52cf614b534f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.572321 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.572295    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjhvn\" (UniqueName: \"kubernetes.io/projected/6809b2de-ff62-492a-92fb-52cf614b534f-kube-api-access-bjhvn\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.703145 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.703036    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:04.891805 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:04.891765    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"]
Jul 29 17:08:04.893195 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:08:04.893151    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6809b2de_ff62_492a_92fb_52cf614b534f.slice/crio-8a10526415b110a88dbf97716d2cfc7e2797d0744756efa40b21d1d6dd9c027f WatchSource:0}: Error finding container 8a10526415b110a88dbf97716d2cfc7e2797d0744756efa40b21d1d6dd9c027f: Status 404 returned error can't find the container with id 8a10526415b110a88dbf97716d2cfc7e2797d0744756efa40b21d1d6dd9c027f
Jul 29 17:08:05.421587 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:05.421555    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" event={"ID":"6809b2de-ff62-492a-92fb-52cf614b534f","Type":"ContainerStarted","Data":"d18eb72aa467dc5f612c052059e69764946e508ce51780b74a58f3aec883444a"}
Jul 29 17:08:05.421972 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:05.421605    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" event={"ID":"6809b2de-ff62-492a-92fb-52cf614b534f","Type":"ContainerStarted","Data":"8a10526415b110a88dbf97716d2cfc7e2797d0744756efa40b21d1d6dd9c027f"}
Jul 29 17:08:10.454084 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:10.454044    2570 generic.go:356] "Generic (PLEG): container finished" podID="6809b2de-ff62-492a-92fb-52cf614b534f" containerID="d18eb72aa467dc5f612c052059e69764946e508ce51780b74a58f3aec883444a" exitCode=0
Jul 29 17:08:10.454084 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:10.454070    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" event={"ID":"6809b2de-ff62-492a-92fb-52cf614b534f","Type":"ContainerDied","Data":"d18eb72aa467dc5f612c052059e69764946e508ce51780b74a58f3aec883444a"}
Jul 29 17:08:11.464189 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:11.464135    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" event={"ID":"6809b2de-ff62-492a-92fb-52cf614b534f","Type":"ContainerStarted","Data":"c9ddd70e849c8640f75d025895db300dd19ac7b6ca001486405955afab53006a"}
Jul 29 17:08:11.485848 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:11.485793    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" podStartSLOduration=2.796700991 podStartE2EDuration="7.485775649s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.689074658s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:08:11.482948887 +0000 UTC m=+1019.972468553" watchObservedRunningTime="2026-07-29 17:08:11.485775649 +0000 UTC m=+1019.975295305"
Jul 29 17:08:12.394912 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:12.394869    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 29 17:08:14.703903 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:14.703853    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:14.704445 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:14.703997    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:08:14.705770 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:14.705730    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 17:08:22.395318 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:22.395261    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 29 17:08:24.704507 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:24.704440    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 17:08:32.395771 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:32.395713    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 29 17:08:34.704643 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:34.704592    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 17:08:42.395210 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:42.395128    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 29 17:08:44.704171 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:44.704121    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 17:08:52.406575 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:52.406539    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:08:52.415141 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:52.415109    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:08:53.939978 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:53.939944    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"]
Jul 29 17:08:53.940605 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:53.940146    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="main" containerID="cri-o://520dcb6e0f40c944f55c96d454ebe0eb01d4bb0899d14b0cf26188d98a611292" gracePeriod=30
Jul 29 17:08:54.703913 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:08:54.703864    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 17:09:00.148153 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:00.148109    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"]
Jul 29 17:09:00.148680 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:00.148467    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629" podUID="f8a77801-f479-434c-874f-4affb131b0c8" containerName="main" containerID="cri-o://56154d75c5d55919727a76ec53cb6f4cc2320fd2b118500f1f25016bd593aba2" gracePeriod=30
Jul 29 17:09:00.148775 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:00.148740    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629" podUID="f8a77801-f479-434c-874f-4affb131b0c8" containerName="tokenizer" containerID="cri-o://13bdecfcec015369838cbe4cda88ec600c9ebbf6054763efb34eb524d97976eb" gracePeriod=30
Jul 29 17:09:00.788215 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:00.788152    2570 generic.go:356] "Generic (PLEG): container finished" podID="f8a77801-f479-434c-874f-4affb131b0c8" containerID="56154d75c5d55919727a76ec53cb6f4cc2320fd2b118500f1f25016bd593aba2" exitCode=0
Jul 29 17:09:00.788215 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:00.788222    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629" event={"ID":"f8a77801-f479-434c-874f-4affb131b0c8","Type":"ContainerDied","Data":"56154d75c5d55919727a76ec53cb6f4cc2320fd2b118500f1f25016bd593aba2"}
Jul 29 17:09:01.623484 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.623407    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:09:01.704890 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.704853    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a77801-f479-434c-874f-4affb131b0c8-tls-certs\") pod \"f8a77801-f479-434c-874f-4affb131b0c8\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") "
Jul 29 17:09:01.705065 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.704902    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-kserve-provision-location\") pod \"f8a77801-f479-434c-874f-4affb131b0c8\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") "
Jul 29 17:09:01.705065 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.704969    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krfch\" (UniqueName: \"kubernetes.io/projected/f8a77801-f479-434c-874f-4affb131b0c8-kube-api-access-krfch\") pod \"f8a77801-f479-434c-874f-4affb131b0c8\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") "
Jul 29 17:09:01.705065 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.705023    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-uds\") pod \"f8a77801-f479-434c-874f-4affb131b0c8\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") "
Jul 29 17:09:01.705065 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.705061    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-cache\") pod \"f8a77801-f479-434c-874f-4affb131b0c8\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") "
Jul 29 17:09:01.705308 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.705087    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-tmp\") pod \"f8a77801-f479-434c-874f-4affb131b0c8\" (UID: \"f8a77801-f479-434c-874f-4affb131b0c8\") "
Jul 29 17:09:01.705448 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.705354    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f8a77801-f479-434c-874f-4affb131b0c8" (UID: "f8a77801-f479-434c-874f-4affb131b0c8"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:09:01.705624 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.705579    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "f8a77801-f479-434c-874f-4affb131b0c8" (UID: "f8a77801-f479-434c-874f-4affb131b0c8"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:09:01.705624 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.705603    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8a77801-f479-434c-874f-4affb131b0c8" (UID: "f8a77801-f479-434c-874f-4affb131b0c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:09:01.705715 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.705609    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "f8a77801-f479-434c-874f-4affb131b0c8" (UID: "f8a77801-f479-434c-874f-4affb131b0c8"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:09:01.707608 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.707581    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a77801-f479-434c-874f-4affb131b0c8-kube-api-access-krfch" (OuterVolumeSpecName: "kube-api-access-krfch") pod "f8a77801-f479-434c-874f-4affb131b0c8" (UID: "f8a77801-f479-434c-874f-4affb131b0c8"). InnerVolumeSpecName "kube-api-access-krfch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:09:01.707717 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.707692    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a77801-f479-434c-874f-4affb131b0c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8a77801-f479-434c-874f-4affb131b0c8" (UID: "f8a77801-f479-434c-874f-4affb131b0c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:09:01.796004 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.795969    2570 generic.go:356] "Generic (PLEG): container finished" podID="f8a77801-f479-434c-874f-4affb131b0c8" containerID="13bdecfcec015369838cbe4cda88ec600c9ebbf6054763efb34eb524d97976eb" exitCode=0
Jul 29 17:09:01.796196 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.796029    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"
Jul 29 17:09:01.796196 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.796058    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629" event={"ID":"f8a77801-f479-434c-874f-4affb131b0c8","Type":"ContainerDied","Data":"13bdecfcec015369838cbe4cda88ec600c9ebbf6054763efb34eb524d97976eb"}
Jul 29 17:09:01.796196 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.796118    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629" event={"ID":"f8a77801-f479-434c-874f-4affb131b0c8","Type":"ContainerDied","Data":"ea147500c6ddf441e5b373e0c7d6ab55b2a90ef3c1a45e61292207430a74d2e0"}
Jul 29 17:09:01.796196 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.796144    2570 scope.go:117] "RemoveContainer" containerID="13bdecfcec015369838cbe4cda88ec600c9ebbf6054763efb34eb524d97976eb"
Jul 29 17:09:01.806324 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.806298    2570 scope.go:117] "RemoveContainer" containerID="56154d75c5d55919727a76ec53cb6f4cc2320fd2b118500f1f25016bd593aba2"
Jul 29 17:09:01.806425 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.806369    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-tmp\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:09:01.806425 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.806389    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a77801-f479-434c-874f-4affb131b0c8-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:09:01.806425 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.806402    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:09:01.806425 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.806416    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krfch\" (UniqueName: \"kubernetes.io/projected/f8a77801-f479-434c-874f-4affb131b0c8-kube-api-access-krfch\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:09:01.806639 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.806428    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-uds\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:09:01.806639 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.806439    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a77801-f479-434c-874f-4affb131b0c8-tokenizer-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:09:01.815339 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.815314    2570 scope.go:117] "RemoveContainer" containerID="0a33e3fdd4b765f0da037564d3ebd1dd811dd38ac1021e73348efa3e00ee61b1"
Jul 29 17:09:01.821686 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.821659    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"]
Jul 29 17:09:01.824593 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.824572    2570 scope.go:117] "RemoveContainer" containerID="13bdecfcec015369838cbe4cda88ec600c9ebbf6054763efb34eb524d97976eb"
Jul 29 17:09:01.824873 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:09:01.824856    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13bdecfcec015369838cbe4cda88ec600c9ebbf6054763efb34eb524d97976eb\": container with ID starting with 13bdecfcec015369838cbe4cda88ec600c9ebbf6054763efb34eb524d97976eb not found: ID does not exist" containerID="13bdecfcec015369838cbe4cda88ec600c9ebbf6054763efb34eb524d97976eb"
Jul 29 17:09:01.824925 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.824882    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13bdecfcec015369838cbe4cda88ec600c9ebbf6054763efb34eb524d97976eb"} err="failed to get container status \"13bdecfcec015369838cbe4cda88ec600c9ebbf6054763efb34eb524d97976eb\": rpc error: code = NotFound desc = could not find container \"13bdecfcec015369838cbe4cda88ec600c9ebbf6054763efb34eb524d97976eb\": container with ID starting with 13bdecfcec015369838cbe4cda88ec600c9ebbf6054763efb34eb524d97976eb not found: ID does not exist"
Jul 29 17:09:01.824925 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.824908    2570 scope.go:117] "RemoveContainer" containerID="56154d75c5d55919727a76ec53cb6f4cc2320fd2b118500f1f25016bd593aba2"
Jul 29 17:09:01.825152 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:09:01.825135    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56154d75c5d55919727a76ec53cb6f4cc2320fd2b118500f1f25016bd593aba2\": container with ID starting with 56154d75c5d55919727a76ec53cb6f4cc2320fd2b118500f1f25016bd593aba2 not found: ID does not exist" containerID="56154d75c5d55919727a76ec53cb6f4cc2320fd2b118500f1f25016bd593aba2"
Jul 29 17:09:01.825246 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.825156    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56154d75c5d55919727a76ec53cb6f4cc2320fd2b118500f1f25016bd593aba2"} err="failed to get container status \"56154d75c5d55919727a76ec53cb6f4cc2320fd2b118500f1f25016bd593aba2\": rpc error: code = NotFound desc = could not find container \"56154d75c5d55919727a76ec53cb6f4cc2320fd2b118500f1f25016bd593aba2\": container with ID starting with 56154d75c5d55919727a76ec53cb6f4cc2320fd2b118500f1f25016bd593aba2 not found: ID does not exist"
Jul 29 17:09:01.825246 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.825194    2570 scope.go:117] "RemoveContainer" containerID="0a33e3fdd4b765f0da037564d3ebd1dd811dd38ac1021e73348efa3e00ee61b1"
Jul 29 17:09:01.825420 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.825400    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-r2629"]
Jul 29 17:09:01.825465 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:09:01.825416    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a33e3fdd4b765f0da037564d3ebd1dd811dd38ac1021e73348efa3e00ee61b1\": container with ID starting with 0a33e3fdd4b765f0da037564d3ebd1dd811dd38ac1021e73348efa3e00ee61b1 not found: ID does not exist" containerID="0a33e3fdd4b765f0da037564d3ebd1dd811dd38ac1021e73348efa3e00ee61b1"
Jul 29 17:09:01.825465 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.825434    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a33e3fdd4b765f0da037564d3ebd1dd811dd38ac1021e73348efa3e00ee61b1"} err="failed to get container status \"0a33e3fdd4b765f0da037564d3ebd1dd811dd38ac1021e73348efa3e00ee61b1\": rpc error: code = NotFound desc = could not find container \"0a33e3fdd4b765f0da037564d3ebd1dd811dd38ac1021e73348efa3e00ee61b1\": container with ID starting with 0a33e3fdd4b765f0da037564d3ebd1dd811dd38ac1021e73348efa3e00ee61b1 not found: ID does not exist"
Jul 29 17:09:01.914771 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:01.914672    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a77801-f479-434c-874f-4affb131b0c8" path="/var/lib/kubelet/pods/f8a77801-f479-434c-874f-4affb131b0c8/volumes"
Jul 29 17:09:04.704537 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:04.704439    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 17:09:09.567413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.567372    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"]
Jul 29 17:09:09.567897 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.567776    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a77801-f479-434c-874f-4affb131b0c8" containerName="tokenizer"
Jul 29 17:09:09.567897 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.567789    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a77801-f479-434c-874f-4affb131b0c8" containerName="tokenizer"
Jul 29 17:09:09.567897 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.567801    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a77801-f479-434c-874f-4affb131b0c8" containerName="storage-initializer"
Jul 29 17:09:09.567897 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.567807    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a77801-f479-434c-874f-4affb131b0c8" containerName="storage-initializer"
Jul 29 17:09:09.567897 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.567820    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a77801-f479-434c-874f-4affb131b0c8" containerName="main"
Jul 29 17:09:09.567897 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.567825    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a77801-f479-434c-874f-4affb131b0c8" containerName="main"
Jul 29 17:09:09.568260 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.567904    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a77801-f479-434c-874f-4affb131b0c8" containerName="main"
Jul 29 17:09:09.568260 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.567919    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a77801-f479-434c-874f-4affb131b0c8" containerName="tokenizer"
Jul 29 17:09:09.571350 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.571322    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.583502 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.583469    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"]
Jul 29 17:09:09.678120 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.678086    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-tmp-dir\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.678340 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.678127    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-model-cache\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.678340 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.678148    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmw2h\" (UniqueName: \"kubernetes.io/projected/c262a233-28cc-4ee8-823b-1194149aa1cc-kube-api-access-jmw2h\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.678340 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.678249    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-home\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.678340 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.678298    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-dshm\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.678340 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.678317    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-kserve-provision-location\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.678511 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.678352    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c262a233-28cc-4ee8-823b-1194149aa1cc-tls-certs\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.778965 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.778926    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c262a233-28cc-4ee8-823b-1194149aa1cc-tls-certs\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.779197 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.779003    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-tmp-dir\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.779197 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.779043    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-model-cache\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.779197 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.779067    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmw2h\" (UniqueName: \"kubernetes.io/projected/c262a233-28cc-4ee8-823b-1194149aa1cc-kube-api-access-jmw2h\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.779197 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.779134    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-home\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.779519 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.779210    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-dshm\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.779519 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.779240    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-kserve-provision-location\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.779621 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.779564    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-model-cache\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.779621 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.779591    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-home\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.779732 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.779687    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-kserve-provision-location\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.779844 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.779820    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-tmp-dir\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.781627 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.781594    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-dshm\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.781821 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.781801    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c262a233-28cc-4ee8-823b-1194149aa1cc-tls-certs\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.786577 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.786552    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmw2h\" (UniqueName: \"kubernetes.io/projected/c262a233-28cc-4ee8-823b-1194149aa1cc-kube-api-access-jmw2h\") pod \"stop-feature-test-kserve-5697d48bb4-9dkjt\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:09.883895 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:09.883801    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:10.038208 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:10.038155    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"]
Jul 29 17:09:10.039606 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:09:10.039578    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc262a233_28cc_4ee8_823b_1194149aa1cc.slice/crio-1c5399591d5026995734f8977a5c42353f959dc94ac6eea5599e4c9a70201cc1 WatchSource:0}: Error finding container 1c5399591d5026995734f8977a5c42353f959dc94ac6eea5599e4c9a70201cc1: Status 404 returned error can't find the container with id 1c5399591d5026995734f8977a5c42353f959dc94ac6eea5599e4c9a70201cc1
Jul 29 17:09:10.858853 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:10.858814    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" event={"ID":"c262a233-28cc-4ee8-823b-1194149aa1cc","Type":"ContainerStarted","Data":"856747bca895941600e48e05c3dff0ef410243ccb34d7c6059d5707f38c21d55"}
Jul 29 17:09:10.859328 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:10.858896    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" event={"ID":"c262a233-28cc-4ee8-823b-1194149aa1cc","Type":"ContainerStarted","Data":"1c5399591d5026995734f8977a5c42353f959dc94ac6eea5599e4c9a70201cc1"}
Jul 29 17:09:14.703990 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:14.703947    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 17:09:14.884747 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:14.884708    2570 generic.go:356] "Generic (PLEG): container finished" podID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerID="856747bca895941600e48e05c3dff0ef410243ccb34d7c6059d5707f38c21d55" exitCode=0
Jul 29 17:09:14.884916 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:14.884760    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" event={"ID":"c262a233-28cc-4ee8-823b-1194149aa1cc","Type":"ContainerDied","Data":"856747bca895941600e48e05c3dff0ef410243ccb34d7c6059d5707f38c21d55"}
Jul 29 17:09:15.892900 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:15.892867    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" event={"ID":"c262a233-28cc-4ee8-823b-1194149aa1cc","Type":"ContainerStarted","Data":"6900f7b329ab68a04c49b18c63306c44b628978abcfb6b3f25d555d07cf90c79"}
Jul 29 17:09:15.915537 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:15.915470    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" podStartSLOduration=2.176614599 podStartE2EDuration="6.915446709s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.73883211s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:09:15.911914875 +0000 UTC m=+1084.401434533" watchObservedRunningTime="2026-07-29 17:09:15.915446709 +0000 UTC m=+1084.404966365"
Jul 29 17:09:19.883968 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:19.883920    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:19.884521 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:19.884005    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:09:19.885593 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:19.885562    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 17:09:24.257017 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.256989    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5697d48bb4-zdsjd_7c695af8-a557-4416-a1f2-99da117161b5/main/0.log"
Jul 29 17:09:24.278530 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.278487    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:09:24.304201 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.304149    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-model-cache\") pod \"7c695af8-a557-4416-a1f2-99da117161b5\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") "
Jul 29 17:09:24.304402 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.304261    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-tmp-dir\") pod \"7c695af8-a557-4416-a1f2-99da117161b5\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") "
Jul 29 17:09:24.304402 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.304294    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-home\") pod \"7c695af8-a557-4416-a1f2-99da117161b5\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") "
Jul 29 17:09:24.304402 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.304328    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-kserve-provision-location\") pod \"7c695af8-a557-4416-a1f2-99da117161b5\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") "
Jul 29 17:09:24.304402 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.304372    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfhqv\" (UniqueName: \"kubernetes.io/projected/7c695af8-a557-4416-a1f2-99da117161b5-kube-api-access-cfhqv\") pod \"7c695af8-a557-4416-a1f2-99da117161b5\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") "
Jul 29 17:09:24.304613 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.304436    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c695af8-a557-4416-a1f2-99da117161b5-tls-certs\") pod \"7c695af8-a557-4416-a1f2-99da117161b5\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") "
Jul 29 17:09:24.304613 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.304465    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-dshm\") pod \"7c695af8-a557-4416-a1f2-99da117161b5\" (UID: \"7c695af8-a557-4416-a1f2-99da117161b5\") "
Jul 29 17:09:24.305493 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.304431    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c695af8-a557-4416-a1f2-99da117161b5" (UID: "7c695af8-a557-4416-a1f2-99da117161b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:09:24.305493 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.305428    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-home" (OuterVolumeSpecName: "home") pod "7c695af8-a557-4416-a1f2-99da117161b5" (UID: "7c695af8-a557-4416-a1f2-99da117161b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:09:24.307956 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.307926    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-dshm" (OuterVolumeSpecName: "dshm") pod "7c695af8-a557-4416-a1f2-99da117161b5" (UID: "7c695af8-a557-4416-a1f2-99da117161b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:09:24.308085 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.307991    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c695af8-a557-4416-a1f2-99da117161b5-kube-api-access-cfhqv" (OuterVolumeSpecName: "kube-api-access-cfhqv") pod "7c695af8-a557-4416-a1f2-99da117161b5" (UID: "7c695af8-a557-4416-a1f2-99da117161b5"). InnerVolumeSpecName "kube-api-access-cfhqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:09:24.318779 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.318733    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c695af8-a557-4416-a1f2-99da117161b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c695af8-a557-4416-a1f2-99da117161b5" (UID: "7c695af8-a557-4416-a1f2-99da117161b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:09:24.323017 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.322984    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c695af8-a557-4416-a1f2-99da117161b5" (UID: "7c695af8-a557-4416-a1f2-99da117161b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:09:24.387822 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.387776    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c695af8-a557-4416-a1f2-99da117161b5" (UID: "7c695af8-a557-4416-a1f2-99da117161b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:09:24.406155 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.406113    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:09:24.406155 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.406149    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:09:24.406418 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.406191    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:09:24.406418 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.406218    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:09:24.406418 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.406234    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfhqv\" (UniqueName: \"kubernetes.io/projected/7c695af8-a557-4416-a1f2-99da117161b5-kube-api-access-cfhqv\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:09:24.406418 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.406247    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c695af8-a557-4416-a1f2-99da117161b5-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:09:24.406418 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.406258    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c695af8-a557-4416-a1f2-99da117161b5-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:09:24.703939 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.703894    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 17:09:24.947086 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.947057    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5697d48bb4-zdsjd_7c695af8-a557-4416-a1f2-99da117161b5/main/0.log"
Jul 29 17:09:24.948562 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.947832    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" event={"ID":"7c695af8-a557-4416-a1f2-99da117161b5","Type":"ContainerDied","Data":"520dcb6e0f40c944f55c96d454ebe0eb01d4bb0899d14b0cf26188d98a611292"}
Jul 29 17:09:24.948562 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.947900    2570 generic.go:356] "Generic (PLEG): container finished" podID="7c695af8-a557-4416-a1f2-99da117161b5" containerID="520dcb6e0f40c944f55c96d454ebe0eb01d4bb0899d14b0cf26188d98a611292" exitCode=137
Jul 29 17:09:24.948562 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.947949    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd" event={"ID":"7c695af8-a557-4416-a1f2-99da117161b5","Type":"ContainerDied","Data":"3000e9dc1531f701398659114aa6c3f821c06ce7b6f46ca25ad2247c8332f3ed"}
Jul 29 17:09:24.948562 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.947888    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"
Jul 29 17:09:24.948562 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.948003    2570 scope.go:117] "RemoveContainer" containerID="520dcb6e0f40c944f55c96d454ebe0eb01d4bb0899d14b0cf26188d98a611292"
Jul 29 17:09:24.960662 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.960639    2570 scope.go:117] "RemoveContainer" containerID="f1d8569ad2626e991f85651368e4d131bd110eea12172e8be18a02bb9dd95558"
Jul 29 17:09:24.972621 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.972591    2570 scope.go:117] "RemoveContainer" containerID="520dcb6e0f40c944f55c96d454ebe0eb01d4bb0899d14b0cf26188d98a611292"
Jul 29 17:09:24.973374 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:09:24.972985    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520dcb6e0f40c944f55c96d454ebe0eb01d4bb0899d14b0cf26188d98a611292\": container with ID starting with 520dcb6e0f40c944f55c96d454ebe0eb01d4bb0899d14b0cf26188d98a611292 not found: ID does not exist" containerID="520dcb6e0f40c944f55c96d454ebe0eb01d4bb0899d14b0cf26188d98a611292"
Jul 29 17:09:24.973374 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.973025    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520dcb6e0f40c944f55c96d454ebe0eb01d4bb0899d14b0cf26188d98a611292"} err="failed to get container status \"520dcb6e0f40c944f55c96d454ebe0eb01d4bb0899d14b0cf26188d98a611292\": rpc error: code = NotFound desc = could not find container \"520dcb6e0f40c944f55c96d454ebe0eb01d4bb0899d14b0cf26188d98a611292\": container with ID starting with 520dcb6e0f40c944f55c96d454ebe0eb01d4bb0899d14b0cf26188d98a611292 not found: ID does not exist"
Jul 29 17:09:24.973374 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.973051    2570 scope.go:117] "RemoveContainer" containerID="f1d8569ad2626e991f85651368e4d131bd110eea12172e8be18a02bb9dd95558"
Jul 29 17:09:24.973374 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:09:24.973359    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d8569ad2626e991f85651368e4d131bd110eea12172e8be18a02bb9dd95558\": container with ID starting with f1d8569ad2626e991f85651368e4d131bd110eea12172e8be18a02bb9dd95558 not found: ID does not exist" containerID="f1d8569ad2626e991f85651368e4d131bd110eea12172e8be18a02bb9dd95558"
Jul 29 17:09:24.973717 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.973388    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d8569ad2626e991f85651368e4d131bd110eea12172e8be18a02bb9dd95558"} err="failed to get container status \"f1d8569ad2626e991f85651368e4d131bd110eea12172e8be18a02bb9dd95558\": rpc error: code = NotFound desc = could not find container \"f1d8569ad2626e991f85651368e4d131bd110eea12172e8be18a02bb9dd95558\": container with ID starting with f1d8569ad2626e991f85651368e4d131bd110eea12172e8be18a02bb9dd95558 not found: ID does not exist"
Jul 29 17:09:24.976421 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.976395    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"]
Jul 29 17:09:24.980257 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:24.980224    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-zdsjd"]
Jul 29 17:09:25.916246 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:25.916205    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c695af8-a557-4416-a1f2-99da117161b5" path="/var/lib/kubelet/pods/7c695af8-a557-4416-a1f2-99da117161b5/volumes"
Jul 29 17:09:29.885098 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:29.885057    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 17:09:34.703809 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:34.703764    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 17:09:39.885259 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:39.885207    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 17:09:44.704259 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:44.704209    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 17:09:49.884590 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:49.884546    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 17:09:54.714070 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:54.714037    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:09:54.722261 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:54.722231    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:09:59.884791 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:09:59.884744    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 17:10:01.971846 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:01.971810    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"]
Jul 29 17:10:01.972345 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:01.972061    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="main" containerID="cri-o://c9ddd70e849c8640f75d025895db300dd19ac7b6ca001486405955afab53006a" gracePeriod=30
Jul 29 17:10:09.884930 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:09.884882    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 17:10:10.574262 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.571036    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"]
Jul 29 17:10:10.574262 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.572050    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="main"
Jul 29 17:10:10.574262 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.572072    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="main"
Jul 29 17:10:10.574262 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.572136    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="storage-initializer"
Jul 29 17:10:10.574262 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.572151    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="storage-initializer"
Jul 29 17:10:10.574262 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.572376    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c695af8-a557-4416-a1f2-99da117161b5" containerName="main"
Jul 29 17:10:10.578402 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.578372    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.582699 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.582669    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 29 17:10:10.584728 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.584701    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"]
Jul 29 17:10:10.643182 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.643134    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.643377 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.643213    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-dshm\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.643377 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.643251    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-home\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.643377 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.643321    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-model-cache\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.643510 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.643385    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/688bbea9-c4d1-4004-b979-a71056c37633-tls-certs\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.643510 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.643429    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgg7j\" (UniqueName: \"kubernetes.io/projected/688bbea9-c4d1-4004-b979-a71056c37633-kube-api-access-cgg7j\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.643510 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.643454    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-tmp-dir\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.744564 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.744521    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-dshm\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.744754 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.744582    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-home\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.744754 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.744620    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-model-cache\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.744754 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.744653    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/688bbea9-c4d1-4004-b979-a71056c37633-tls-certs\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.744754 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.744688    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgg7j\" (UniqueName: \"kubernetes.io/projected/688bbea9-c4d1-4004-b979-a71056c37633-kube-api-access-cgg7j\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.744754 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.744716    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-tmp-dir\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.745031 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.744768    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.745083 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.745065    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-model-cache\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.745203 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.745083    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-home\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.745309 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.745233    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.745475 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.745448    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-tmp-dir\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.747035 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.747009    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-dshm\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.747323 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.747303    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/688bbea9-c4d1-4004-b979-a71056c37633-tls-certs\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.753280 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.753247    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgg7j\" (UniqueName: \"kubernetes.io/projected/688bbea9-c4d1-4004-b979-a71056c37633-kube-api-access-cgg7j\") pod \"custom-route-timeout-test-kserve-fcb85f744-42x5r\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:10.891851 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:10.891743    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:11.047070 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:11.047040    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"]
Jul 29 17:10:11.049064 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:10:11.049035    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688bbea9_c4d1_4004_b979_a71056c37633.slice/crio-bcec964b12305a072173a4f0f31d55438c56a3fdaadf58ec6c24c91c49fbe59c WatchSource:0}: Error finding container bcec964b12305a072173a4f0f31d55438c56a3fdaadf58ec6c24c91c49fbe59c: Status 404 returned error can't find the container with id bcec964b12305a072173a4f0f31d55438c56a3fdaadf58ec6c24c91c49fbe59c
Jul 29 17:10:11.248782 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:11.248674    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" event={"ID":"688bbea9-c4d1-4004-b979-a71056c37633","Type":"ContainerStarted","Data":"52a13a27a9f0f34f6b9c2af42cf2f9f3b9e62f0366c828413d4055cd5f6cefba"}
Jul 29 17:10:11.248782 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:11.248745    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" event={"ID":"688bbea9-c4d1-4004-b979-a71056c37633","Type":"ContainerStarted","Data":"bcec964b12305a072173a4f0f31d55438c56a3fdaadf58ec6c24c91c49fbe59c"}
Jul 29 17:10:16.287616 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:16.287572    2570 generic.go:356] "Generic (PLEG): container finished" podID="688bbea9-c4d1-4004-b979-a71056c37633" containerID="52a13a27a9f0f34f6b9c2af42cf2f9f3b9e62f0366c828413d4055cd5f6cefba" exitCode=0
Jul 29 17:10:16.288185 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:16.287648    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" event={"ID":"688bbea9-c4d1-4004-b979-a71056c37633","Type":"ContainerDied","Data":"52a13a27a9f0f34f6b9c2af42cf2f9f3b9e62f0366c828413d4055cd5f6cefba"}
Jul 29 17:10:17.297819 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:17.297782    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" event={"ID":"688bbea9-c4d1-4004-b979-a71056c37633","Type":"ContainerStarted","Data":"1a287bb81f905afc89cf8389a862eb8e877f814ef62b8467369a609c1831a4c4"}
Jul 29 17:10:17.322070 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:17.322000    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" podStartSLOduration=2.730402499 podStartE2EDuration="7.321973021s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.591570522s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:10:17.317328577 +0000 UTC m=+1145.806848246" watchObservedRunningTime="2026-07-29 17:10:17.321973021 +0000 UTC m=+1145.811492680"
Jul 29 17:10:19.884933 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:19.884878    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 17:10:20.891960 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:20.891911    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:20.892482 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:20.891998    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:10:20.893877 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:20.893841    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 29 17:10:29.884259 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:29.884211    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 17:10:30.892444 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:30.892387    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 29 17:10:32.402231 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.402201    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b_6809b2de-ff62-492a-92fb-52cf614b534f/main/0.log"
Jul 29 17:10:32.402667 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.402642    2570 generic.go:356] "Generic (PLEG): container finished" podID="6809b2de-ff62-492a-92fb-52cf614b534f" containerID="c9ddd70e849c8640f75d025895db300dd19ac7b6ca001486405955afab53006a" exitCode=137
Jul 29 17:10:32.402740 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.402716    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" event={"ID":"6809b2de-ff62-492a-92fb-52cf614b534f","Type":"ContainerDied","Data":"c9ddd70e849c8640f75d025895db300dd19ac7b6ca001486405955afab53006a"}
Jul 29 17:10:32.418840 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.418812    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b_6809b2de-ff62-492a-92fb-52cf614b534f/main/0.log"
Jul 29 17:10:32.421212 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.421147    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:10:32.556755 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.556712    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-tmp-dir\") pod \"6809b2de-ff62-492a-92fb-52cf614b534f\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") "
Jul 29 17:10:32.556755 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.556761    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjhvn\" (UniqueName: \"kubernetes.io/projected/6809b2de-ff62-492a-92fb-52cf614b534f-kube-api-access-bjhvn\") pod \"6809b2de-ff62-492a-92fb-52cf614b534f\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") "
Jul 29 17:10:32.557026 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.556782    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-model-cache\") pod \"6809b2de-ff62-492a-92fb-52cf614b534f\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") "
Jul 29 17:10:32.557026 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.556806    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-home\") pod \"6809b2de-ff62-492a-92fb-52cf614b534f\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") "
Jul 29 17:10:32.557026 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.556846    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-dshm\") pod \"6809b2de-ff62-492a-92fb-52cf614b534f\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") "
Jul 29 17:10:32.557026 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.556895    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6809b2de-ff62-492a-92fb-52cf614b534f-tls-certs\") pod \"6809b2de-ff62-492a-92fb-52cf614b534f\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") "
Jul 29 17:10:32.557026 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.556928    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-kserve-provision-location\") pod \"6809b2de-ff62-492a-92fb-52cf614b534f\" (UID: \"6809b2de-ff62-492a-92fb-52cf614b534f\") "
Jul 29 17:10:32.557673 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.557457    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6809b2de-ff62-492a-92fb-52cf614b534f" (UID: "6809b2de-ff62-492a-92fb-52cf614b534f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:10:32.558014 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.557967    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:10:32.558141 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.558000    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-home" (OuterVolumeSpecName: "home") pod "6809b2de-ff62-492a-92fb-52cf614b534f" (UID: "6809b2de-ff62-492a-92fb-52cf614b534f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:10:32.575365 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.560622    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6809b2de-ff62-492a-92fb-52cf614b534f-kube-api-access-bjhvn" (OuterVolumeSpecName: "kube-api-access-bjhvn") pod "6809b2de-ff62-492a-92fb-52cf614b534f" (UID: "6809b2de-ff62-492a-92fb-52cf614b534f"). InnerVolumeSpecName "kube-api-access-bjhvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:10:32.575365 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.560661    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6809b2de-ff62-492a-92fb-52cf614b534f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6809b2de-ff62-492a-92fb-52cf614b534f" (UID: "6809b2de-ff62-492a-92fb-52cf614b534f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:10:32.577931 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.577878    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6809b2de-ff62-492a-92fb-52cf614b534f" (UID: "6809b2de-ff62-492a-92fb-52cf614b534f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:10:32.578113 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.578088    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-dshm" (OuterVolumeSpecName: "dshm") pod "6809b2de-ff62-492a-92fb-52cf614b534f" (UID: "6809b2de-ff62-492a-92fb-52cf614b534f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:10:32.595385 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.595306    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6809b2de-ff62-492a-92fb-52cf614b534f" (UID: "6809b2de-ff62-492a-92fb-52cf614b534f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:10:32.658772 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.658740    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:10:32.658772 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.658769    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjhvn\" (UniqueName: \"kubernetes.io/projected/6809b2de-ff62-492a-92fb-52cf614b534f-kube-api-access-bjhvn\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:10:32.658772 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.658778    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:10:32.659019 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.658788    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:10:32.659019 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.658796    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6809b2de-ff62-492a-92fb-52cf614b534f-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:10:32.659019 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:32.658804    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6809b2de-ff62-492a-92fb-52cf614b534f-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:10:33.412303 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:33.412223    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b_6809b2de-ff62-492a-92fb-52cf614b534f/main/0.log"
Jul 29 17:10:33.412810 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:33.412780    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b" event={"ID":"6809b2de-ff62-492a-92fb-52cf614b534f","Type":"ContainerDied","Data":"8a10526415b110a88dbf97716d2cfc7e2797d0744756efa40b21d1d6dd9c027f"}
Jul 29 17:10:33.412888 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:33.412815    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"
Jul 29 17:10:33.412888 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:33.412857    2570 scope.go:117] "RemoveContainer" containerID="c9ddd70e849c8640f75d025895db300dd19ac7b6ca001486405955afab53006a"
Jul 29 17:10:33.423251 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:33.423223    2570 scope.go:117] "RemoveContainer" containerID="d18eb72aa467dc5f612c052059e69764946e508ce51780b74a58f3aec883444a"
Jul 29 17:10:33.441324 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:33.441295    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"]
Jul 29 17:10:33.445288 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:33.445220    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7b66b5454d7vp4b"]
Jul 29 17:10:33.916200 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:33.916141    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" path="/var/lib/kubelet/pods/6809b2de-ff62-492a-92fb-52cf614b534f/volumes"
Jul 29 17:10:39.884324 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:39.884268    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 17:10:40.893183 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:40.893121    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 29 17:10:49.884514 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:49.884474    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 17:10:50.892889 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:50.892778    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 29 17:10:59.896586 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:59.895108    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:10:59.914020 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:10:59.913992    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:11:00.892574 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:00.892517    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 29 17:11:01.147606 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:01.147519    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"]
Jul 29 17:11:01.589284 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:01.589233    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="main" containerID="cri-o://6900f7b329ab68a04c49b18c63306c44b628978abcfb6b3f25d555d07cf90c79" gracePeriod=30
Jul 29 17:11:03.449480 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.449443    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb"]
Jul 29 17:11:03.449923 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.449892    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="main"
Jul 29 17:11:03.449923 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.449907    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="main"
Jul 29 17:11:03.449923 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.449924    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="storage-initializer"
Jul 29 17:11:03.450036 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.449930    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="storage-initializer"
Jul 29 17:11:03.450036 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.449996    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6809b2de-ff62-492a-92fb-52cf614b534f" containerName="main"
Jul 29 17:11:03.454475 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.454452    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb"
Jul 29 17:11:03.461556 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.461513    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb"]
Jul 29 17:11:03.535587 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.535552    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7860416e-8673-4117-90b2-0aff71370ac8-cert\") pod \"llmisvc-controller-manager-d6cdfdb88-kq6gb\" (UID: \"7860416e-8673-4117-90b2-0aff71370ac8\") " pod="kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb"
Jul 29 17:11:03.535587 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.535601    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2rkw\" (UniqueName: \"kubernetes.io/projected/7860416e-8673-4117-90b2-0aff71370ac8-kube-api-access-l2rkw\") pod \"llmisvc-controller-manager-d6cdfdb88-kq6gb\" (UID: \"7860416e-8673-4117-90b2-0aff71370ac8\") " pod="kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb"
Jul 29 17:11:03.636801 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.636766    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7860416e-8673-4117-90b2-0aff71370ac8-cert\") pod \"llmisvc-controller-manager-d6cdfdb88-kq6gb\" (UID: \"7860416e-8673-4117-90b2-0aff71370ac8\") " pod="kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb"
Jul 29 17:11:03.636977 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.636819    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2rkw\" (UniqueName: \"kubernetes.io/projected/7860416e-8673-4117-90b2-0aff71370ac8-kube-api-access-l2rkw\") pod \"llmisvc-controller-manager-d6cdfdb88-kq6gb\" (UID: \"7860416e-8673-4117-90b2-0aff71370ac8\") " pod="kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb"
Jul 29 17:11:03.639403 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.639373    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7860416e-8673-4117-90b2-0aff71370ac8-cert\") pod \"llmisvc-controller-manager-d6cdfdb88-kq6gb\" (UID: \"7860416e-8673-4117-90b2-0aff71370ac8\") " pod="kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb"
Jul 29 17:11:03.647933 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.647894    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2rkw\" (UniqueName: \"kubernetes.io/projected/7860416e-8673-4117-90b2-0aff71370ac8-kube-api-access-l2rkw\") pod \"llmisvc-controller-manager-d6cdfdb88-kq6gb\" (UID: \"7860416e-8673-4117-90b2-0aff71370ac8\") " pod="kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb"
Jul 29 17:11:03.766500 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.766472    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb"
Jul 29 17:11:03.913011 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:11:03.912980    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7860416e_8673_4117_90b2_0aff71370ac8.slice/crio-fee55acc0fd5aabb3582332fcbba618eeaa3da0552ec1dc465fd21795a435e7d WatchSource:0}: Error finding container fee55acc0fd5aabb3582332fcbba618eeaa3da0552ec1dc465fd21795a435e7d: Status 404 returned error can't find the container with id fee55acc0fd5aabb3582332fcbba618eeaa3da0552ec1dc465fd21795a435e7d
Jul 29 17:11:03.914528 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.914506    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 17:11:03.915641 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:03.915621    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb"]
Jul 29 17:11:04.606266 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:04.606228    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb" event={"ID":"7860416e-8673-4117-90b2-0aff71370ac8","Type":"ContainerStarted","Data":"440daf51fe77400b81865061850e62ec7fabb8495abf23fe8e019130835ffb37"}
Jul 29 17:11:04.606672 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:04.606293    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb" event={"ID":"7860416e-8673-4117-90b2-0aff71370ac8","Type":"ContainerStarted","Data":"fee55acc0fd5aabb3582332fcbba618eeaa3da0552ec1dc465fd21795a435e7d"}
Jul 29 17:11:04.606672 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:04.606443    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb"
Jul 29 17:11:04.626073 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:04.626018    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb" podStartSLOduration=1.152005582 podStartE2EDuration="1.625999385s" totalImagesPullingTime="473.993803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:11:04.623877852 +0000 UTC m=+1193.113397509" watchObservedRunningTime="2026-07-29 17:11:04.625999385 +0000 UTC m=+1193.115519041"
Jul 29 17:11:10.892914 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:10.892865    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 29 17:11:11.941216 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:11.941180    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:11:11.943619 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:11.943599    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:11:20.892438 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:20.892396    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 29 17:11:30.892723 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:30.892677    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 29 17:11:31.765946 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:31.765911    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5697d48bb4-9dkjt_c262a233-28cc-4ee8-823b-1194149aa1cc/main/0.log"
Jul 29 17:11:31.766383 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:31.766355    2570 generic.go:356] "Generic (PLEG): container finished" podID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerID="6900f7b329ab68a04c49b18c63306c44b628978abcfb6b3f25d555d07cf90c79" exitCode=137
Jul 29 17:11:31.766481 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:31.766428    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" event={"ID":"c262a233-28cc-4ee8-823b-1194149aa1cc","Type":"ContainerDied","Data":"6900f7b329ab68a04c49b18c63306c44b628978abcfb6b3f25d555d07cf90c79"}
Jul 29 17:11:31.901553 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:31.901527    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5697d48bb4-9dkjt_c262a233-28cc-4ee8-823b-1194149aa1cc/main/0.log"
Jul 29 17:11:31.904925 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:31.904902    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:11:32.103280 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.103247    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-model-cache\") pod \"c262a233-28cc-4ee8-823b-1194149aa1cc\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") "
Jul 29 17:11:32.103280 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.103284    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-dshm\") pod \"c262a233-28cc-4ee8-823b-1194149aa1cc\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") "
Jul 29 17:11:32.103651 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.103329    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmw2h\" (UniqueName: \"kubernetes.io/projected/c262a233-28cc-4ee8-823b-1194149aa1cc-kube-api-access-jmw2h\") pod \"c262a233-28cc-4ee8-823b-1194149aa1cc\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") "
Jul 29 17:11:32.103651 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.103395    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-tmp-dir\") pod \"c262a233-28cc-4ee8-823b-1194149aa1cc\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") "
Jul 29 17:11:32.103651 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.103423    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-home\") pod \"c262a233-28cc-4ee8-823b-1194149aa1cc\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") "
Jul 29 17:11:32.103651 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.103446    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-kserve-provision-location\") pod \"c262a233-28cc-4ee8-823b-1194149aa1cc\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") "
Jul 29 17:11:32.103651 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.103479    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c262a233-28cc-4ee8-823b-1194149aa1cc-tls-certs\") pod \"c262a233-28cc-4ee8-823b-1194149aa1cc\" (UID: \"c262a233-28cc-4ee8-823b-1194149aa1cc\") "
Jul 29 17:11:32.103651 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.103609    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "c262a233-28cc-4ee8-823b-1194149aa1cc" (UID: "c262a233-28cc-4ee8-823b-1194149aa1cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:11:32.103981 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.103875    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:11:32.104568 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.104541    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-home" (OuterVolumeSpecName: "home") pod "c262a233-28cc-4ee8-823b-1194149aa1cc" (UID: "c262a233-28cc-4ee8-823b-1194149aa1cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:11:32.106094 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.106066    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c262a233-28cc-4ee8-823b-1194149aa1cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c262a233-28cc-4ee8-823b-1194149aa1cc" (UID: "c262a233-28cc-4ee8-823b-1194149aa1cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:11:32.106238 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.106113    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-dshm" (OuterVolumeSpecName: "dshm") pod "c262a233-28cc-4ee8-823b-1194149aa1cc" (UID: "c262a233-28cc-4ee8-823b-1194149aa1cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:11:32.106389 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.106363    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c262a233-28cc-4ee8-823b-1194149aa1cc-kube-api-access-jmw2h" (OuterVolumeSpecName: "kube-api-access-jmw2h") pod "c262a233-28cc-4ee8-823b-1194149aa1cc" (UID: "c262a233-28cc-4ee8-823b-1194149aa1cc"). InnerVolumeSpecName "kube-api-access-jmw2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:11:32.117095 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.117050    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c262a233-28cc-4ee8-823b-1194149aa1cc" (UID: "c262a233-28cc-4ee8-823b-1194149aa1cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:11:32.173464 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.173354    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c262a233-28cc-4ee8-823b-1194149aa1cc" (UID: "c262a233-28cc-4ee8-823b-1194149aa1cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:11:32.205286 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.205244    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:11:32.205286 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.205281    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:11:32.205286 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.205293    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c262a233-28cc-4ee8-823b-1194149aa1cc-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:11:32.205510 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.205302    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:11:32.205510 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.205311    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmw2h\" (UniqueName: \"kubernetes.io/projected/c262a233-28cc-4ee8-823b-1194149aa1cc-kube-api-access-jmw2h\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:11:32.205510 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.205318    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c262a233-28cc-4ee8-823b-1194149aa1cc-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:11:32.774236 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.774209    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5697d48bb4-9dkjt_c262a233-28cc-4ee8-823b-1194149aa1cc/main/0.log"
Jul 29 17:11:32.776412 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.774664    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"
Jul 29 17:11:32.776412 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.774677    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt" event={"ID":"c262a233-28cc-4ee8-823b-1194149aa1cc","Type":"ContainerDied","Data":"1c5399591d5026995734f8977a5c42353f959dc94ac6eea5599e4c9a70201cc1"}
Jul 29 17:11:32.776412 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.774761    2570 scope.go:117] "RemoveContainer" containerID="6900f7b329ab68a04c49b18c63306c44b628978abcfb6b3f25d555d07cf90c79"
Jul 29 17:11:32.784314 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.784294    2570 scope.go:117] "RemoveContainer" containerID="856747bca895941600e48e05c3dff0ef410243ccb34d7c6059d5707f38c21d55"
Jul 29 17:11:32.802540 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.802497    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"]
Jul 29 17:11:32.805265 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:32.805241    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5697d48bb4-9dkjt"]
Jul 29 17:11:33.915688 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:33.915647    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" path="/var/lib/kubelet/pods/c262a233-28cc-4ee8-823b-1194149aa1cc/volumes"
Jul 29 17:11:35.615115 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:35.615085    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-d6cdfdb88-kq6gb"
Jul 29 17:11:35.661749 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:35.661710    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-84b6684489-mlktn"]
Jul 29 17:11:35.662003 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:35.661968    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-84b6684489-mlktn" podUID="5faf75f6-2825-4b36-9e84-0e52a96b2085" containerName="manager" containerID="cri-o://cef85d7a2befbd221322f446a977a8bd10a59985180a6f37e36a00305bb1cb29" gracePeriod=30
Jul 29 17:11:35.797042 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:35.797011    2570 generic.go:356] "Generic (PLEG): container finished" podID="5faf75f6-2825-4b36-9e84-0e52a96b2085" containerID="cef85d7a2befbd221322f446a977a8bd10a59985180a6f37e36a00305bb1cb29" exitCode=0
Jul 29 17:11:35.797237 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:35.797081    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84b6684489-mlktn" event={"ID":"5faf75f6-2825-4b36-9e84-0e52a96b2085","Type":"ContainerDied","Data":"cef85d7a2befbd221322f446a977a8bd10a59985180a6f37e36a00305bb1cb29"}
Jul 29 17:11:35.930878 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:35.930854    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84b6684489-mlktn"
Jul 29 17:11:35.931872 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:35.931855    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5faf75f6-2825-4b36-9e84-0e52a96b2085-cert\") pod \"5faf75f6-2825-4b36-9e84-0e52a96b2085\" (UID: \"5faf75f6-2825-4b36-9e84-0e52a96b2085\") "
Jul 29 17:11:35.931965 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:35.931911    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb9sr\" (UniqueName: \"kubernetes.io/projected/5faf75f6-2825-4b36-9e84-0e52a96b2085-kube-api-access-mb9sr\") pod \"5faf75f6-2825-4b36-9e84-0e52a96b2085\" (UID: \"5faf75f6-2825-4b36-9e84-0e52a96b2085\") "
Jul 29 17:11:35.933997 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:35.933968    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5faf75f6-2825-4b36-9e84-0e52a96b2085-cert" (OuterVolumeSpecName: "cert") pod "5faf75f6-2825-4b36-9e84-0e52a96b2085" (UID: "5faf75f6-2825-4b36-9e84-0e52a96b2085"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:11:35.934101 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:35.934017    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5faf75f6-2825-4b36-9e84-0e52a96b2085-kube-api-access-mb9sr" (OuterVolumeSpecName: "kube-api-access-mb9sr") pod "5faf75f6-2825-4b36-9e84-0e52a96b2085" (UID: "5faf75f6-2825-4b36-9e84-0e52a96b2085"). InnerVolumeSpecName "kube-api-access-mb9sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:11:36.032541 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:36.032498    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb9sr\" (UniqueName: \"kubernetes.io/projected/5faf75f6-2825-4b36-9e84-0e52a96b2085-kube-api-access-mb9sr\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:11:36.032541 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:36.032532    2570 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5faf75f6-2825-4b36-9e84-0e52a96b2085-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:11:36.804149 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:36.804117    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84b6684489-mlktn"
Jul 29 17:11:36.804149 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:36.804136    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84b6684489-mlktn" event={"ID":"5faf75f6-2825-4b36-9e84-0e52a96b2085","Type":"ContainerDied","Data":"f0f19161974e131f92a27daeba882a0987dd4f1bc18a8d5094335b6ea2556354"}
Jul 29 17:11:36.804666 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:36.804232    2570 scope.go:117] "RemoveContainer" containerID="cef85d7a2befbd221322f446a977a8bd10a59985180a6f37e36a00305bb1cb29"
Jul 29 17:11:36.827307 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:36.827276    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-84b6684489-mlktn"]
Jul 29 17:11:36.831474 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:36.831447    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-84b6684489-mlktn"]
Jul 29 17:11:37.915622 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:37.915584    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5faf75f6-2825-4b36-9e84-0e52a96b2085" path="/var/lib/kubelet/pods/5faf75f6-2825-4b36-9e84-0e52a96b2085/volumes"
Jul 29 17:11:40.893051 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:40.893000    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 29 17:11:50.893133 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:11:50.893024    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 29 17:12:00.902344 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:00.902309    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:12:00.910772 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:00.910735    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:12:04.884371 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.884292    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"]
Jul 29 17:12:04.884805 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.884667    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="main"
Jul 29 17:12:04.884805 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.884678    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="main"
Jul 29 17:12:04.884805 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.884695    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5faf75f6-2825-4b36-9e84-0e52a96b2085" containerName="manager"
Jul 29 17:12:04.884805 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.884702    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5faf75f6-2825-4b36-9e84-0e52a96b2085" containerName="manager"
Jul 29 17:12:04.884805 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.884709    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="storage-initializer"
Jul 29 17:12:04.884805 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.884715    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="storage-initializer"
Jul 29 17:12:04.884805 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.884781    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5faf75f6-2825-4b36-9e84-0e52a96b2085" containerName="manager"
Jul 29 17:12:04.884805 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.884794    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c262a233-28cc-4ee8-823b-1194149aa1cc" containerName="main"
Jul 29 17:12:04.889019 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.888999    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:04.891237 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.891213    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 29 17:12:04.891237 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.891213    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-b46wm\""
Jul 29 17:12:04.898441 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.898412    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"]
Jul 29 17:12:04.987098 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.987066    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-workload-certs\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:04.987098 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.987101    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c2292a-edbb-4c85-8044-192149996162-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:04.987378 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.987128    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c2292a-edbb-4c85-8044-192149996162-istio-podinfo\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:04.987378 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.987177    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nmvq\" (UniqueName: \"kubernetes.io/projected/35c2292a-edbb-4c85-8044-192149996162-kube-api-access-6nmvq\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:04.987378 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.987265    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-workload-socket\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:04.987378 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.987295    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-credential-socket\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:04.987378 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.987311    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-istio-data\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:04.987378 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.987332    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-istio-envoy\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:04.987378 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:04.987356    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c2292a-edbb-4c85-8044-192149996162-istio-token\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.088575 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.088535    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-workload-socket\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.088575 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.088575    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-credential-socket\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.088854 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.088593    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-istio-data\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.088854 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.088611    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-istio-envoy\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.088854 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.088633    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c2292a-edbb-4c85-8044-192149996162-istio-token\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.088854 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.088704    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-workload-certs\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.088854 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.088732    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c2292a-edbb-4c85-8044-192149996162-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.088854 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.088763    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c2292a-edbb-4c85-8044-192149996162-istio-podinfo\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.088854 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.088788    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nmvq\" (UniqueName: \"kubernetes.io/projected/35c2292a-edbb-4c85-8044-192149996162-kube-api-access-6nmvq\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.089262 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.088999    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-workload-socket\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.089262 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.089126    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-istio-data\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.089262 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.089224    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-workload-certs\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.089413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.089333    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-credential-socket\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.089452 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.089417    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c2292a-edbb-4c85-8044-192149996162-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.091115 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.091087    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c2292a-edbb-4c85-8044-192149996162-istio-envoy\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.091467 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.091438    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c2292a-edbb-4c85-8044-192149996162-istio-podinfo\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.096243 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.096219    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nmvq\" (UniqueName: \"kubernetes.io/projected/35c2292a-edbb-4c85-8044-192149996162-kube-api-access-6nmvq\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.096342 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.096274    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c2292a-edbb-4c85-8044-192149996162-istio-token\") pod \"router-gateway-2-openshift-default-7b76d7c4bf-4txdd\" (UID: \"35c2292a-edbb-4c85-8044-192149996162\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.200873 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.200776    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:05.383478 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.383450    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"]
Jul 29 17:12:05.384783 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:12:05.384756    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c2292a_edbb_4c85_8044_192149996162.slice/crio-fb18c22fd6ec2c51b3ebd493ae36842065550605b0415f9c82cc7c594f71de56 WatchSource:0}: Error finding container fb18c22fd6ec2c51b3ebd493ae36842065550605b0415f9c82cc7c594f71de56: Status 404 returned error can't find the container with id fb18c22fd6ec2c51b3ebd493ae36842065550605b0415f9c82cc7c594f71de56
Jul 29 17:12:05.984300 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:05.984269    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd" event={"ID":"35c2292a-edbb-4c85-8044-192149996162","Type":"ContainerStarted","Data":"fb18c22fd6ec2c51b3ebd493ae36842065550605b0415f9c82cc7c594f71de56"}
Jul 29 17:12:07.483317 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:07.483277    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"]
Jul 29 17:12:07.483831 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:07.483644    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="main" containerID="cri-o://1a287bb81f905afc89cf8389a862eb8e877f814ef62b8467369a609c1831a4c4" gracePeriod=30
Jul 29 17:12:08.595463 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:08.595427    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 29 17:12:08.595770 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:08.595523    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 29 17:12:08.595770 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:08.595558    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 29 17:12:09.004943 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:09.004910    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd" event={"ID":"35c2292a-edbb-4c85-8044-192149996162","Type":"ContainerStarted","Data":"abaf0ef7c2e2ea0caf4149202be1b688894f945f8f042187ae2a1762839b2377"}
Jul 29 17:12:09.025810 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:09.025745    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd" podStartSLOduration=1.8173187560000001 podStartE2EDuration="5.025724369s" totalImagesPullingTime="3.208405613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:12:09.025034418 +0000 UTC m=+1257.514554074" watchObservedRunningTime="2026-07-29 17:12:09.025724369 +0000 UTC m=+1257.515244026"
Jul 29 17:12:09.201717 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:09.201662    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:09.204271 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:09.203843    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd" podUID="35c2292a-edbb-4c85-8044-192149996162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 29 17:12:10.201741 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:10.201699    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd" podUID="35c2292a-edbb-4c85-8044-192149996162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 29 17:12:11.201559 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:11.201514    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd" podUID="35c2292a-edbb-4c85-8044-192149996162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 29 17:12:12.201656 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:12.201611    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd" podUID="35c2292a-edbb-4c85-8044-192149996162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 29 17:12:13.201560 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:13.201515    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd" podUID="35c2292a-edbb-4c85-8044-192149996162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 29 17:12:14.202000 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:14.201954    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd" podUID="35c2292a-edbb-4c85-8044-192149996162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 29 17:12:15.205738 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:15.205699    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:15.206140 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:15.206100    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:15.207034 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:15.207013    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7b76d7c4bf-4txdd"
Jul 29 17:12:24.409461 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.409424    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"]
Jul 29 17:12:24.416078 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.416054    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.418620 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.418594    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 29 17:12:24.418818 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.418630    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-tsxcz\""
Jul 29 17:12:24.426055 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.426016    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"]
Jul 29 17:12:24.433855 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.433811    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"]
Jul 29 17:12:24.438104 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.438081    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.450500 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.450472    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"]
Jul 29 17:12:24.474189 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.474137    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-home\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.474189 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.474193    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.474423 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.474215    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7sns\" (UniqueName: \"kubernetes.io/projected/27c541a7-5524-4b02-858e-030f4602f7b8-kube-api-access-w7sns\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.474423 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.474231    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-dshm\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.474423 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.474271    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.474423 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.474289    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.474423 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.474305    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.474423 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.474321    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-tls-certs\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.474423 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.474342    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-home\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.474670 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.474426    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c541a7-5524-4b02-858e-030f4602f7b8-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.474670 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.474489    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7qdd\" (UniqueName: \"kubernetes.io/projected/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-kube-api-access-s7qdd\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.474670 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.474523    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.474670 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.474547    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.474670 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.474569    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-model-cache\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.575673 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.575634    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.575673 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.575678    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.575925 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.575709    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-model-cache\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.575925 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.575746    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-home\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.575925 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.575767    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.575925 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.575793    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7sns\" (UniqueName: \"kubernetes.io/projected/27c541a7-5524-4b02-858e-030f4602f7b8-kube-api-access-w7sns\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.575925 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.575817    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-dshm\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.575925 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.575871    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.575925 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.575893    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.575925 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.575916    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.576364 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.575939    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-tls-certs\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.576364 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.575973    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-home\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.576364 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.576006    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c541a7-5524-4b02-858e-030f4602f7b8-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.576364 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.576054    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7qdd\" (UniqueName: \"kubernetes.io/projected/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-kube-api-access-s7qdd\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.576364 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.576140    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.576364 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.576140    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.576364 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.576213    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-home\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.576364 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.576224    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.576364 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.576288    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.576821 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.576468    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-home\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.576821 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.576517    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.576821 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.576770    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-model-cache\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.578670 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.578645    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-dshm\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.578800 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.578738    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.578887 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.578825    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-tls-certs\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.579026 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.579006    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c541a7-5524-4b02-858e-030f4602f7b8-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.583520 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.583501    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7sns\" (UniqueName: \"kubernetes.io/projected/27c541a7-5524-4b02-858e-030f4602f7b8-kube-api-access-w7sns\") pod \"router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.583604 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.583556    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7qdd\" (UniqueName: \"kubernetes.io/projected/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-kube-api-access-s7qdd\") pod \"router-with-refs-pd-test-kserve-fb85fd64b-9bs98\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.729770 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.729683    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:24.751282 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.751247    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:24.914336 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.914305    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"]
Jul 29 17:12:24.917884 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:12:24.917855    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46573f8c_00e9_4c4a_ba78_9054cdd07c5a.slice/crio-71db65f23118c42f2d134d88cb92b7e44cf3a7da3b1dc5ee6f4c0a2948e1d5f8 WatchSource:0}: Error finding container 71db65f23118c42f2d134d88cb92b7e44cf3a7da3b1dc5ee6f4c0a2948e1d5f8: Status 404 returned error can't find the container with id 71db65f23118c42f2d134d88cb92b7e44cf3a7da3b1dc5ee6f4c0a2948e1d5f8
Jul 29 17:12:24.934942 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:24.934918    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"]
Jul 29 17:12:24.936654 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:12:24.936629    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c541a7_5524_4b02_858e_030f4602f7b8.slice/crio-3ac683b1de95315e12851abd8685fea6fe66cf51260cf031e66e092d9e5cf18b WatchSource:0}: Error finding container 3ac683b1de95315e12851abd8685fea6fe66cf51260cf031e66e092d9e5cf18b: Status 404 returned error can't find the container with id 3ac683b1de95315e12851abd8685fea6fe66cf51260cf031e66e092d9e5cf18b
Jul 29 17:12:25.107073 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:25.107032    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" event={"ID":"46573f8c-00e9-4c4a-ba78-9054cdd07c5a","Type":"ContainerStarted","Data":"71db65f23118c42f2d134d88cb92b7e44cf3a7da3b1dc5ee6f4c0a2948e1d5f8"}
Jul 29 17:12:25.108916 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:25.108883    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" event={"ID":"27c541a7-5524-4b02-858e-030f4602f7b8","Type":"ContainerStarted","Data":"6b4aaa460737693d6f42bfb346e82aae2a66750f1cd40c03f75ac49c1aec78a9"}
Jul 29 17:12:25.109054 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:25.108942    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" event={"ID":"27c541a7-5524-4b02-858e-030f4602f7b8","Type":"ContainerStarted","Data":"3ac683b1de95315e12851abd8685fea6fe66cf51260cf031e66e092d9e5cf18b"}
Jul 29 17:12:26.116454 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:26.116412    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" event={"ID":"46573f8c-00e9-4c4a-ba78-9054cdd07c5a","Type":"ContainerStarted","Data":"c8281c88aa53d5a0e9cef436df858cba8e4284be91c22883bf8d22b10fc1428e"}
Jul 29 17:12:26.116904 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:26.116848    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:27.125032 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:27.124980    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" event={"ID":"46573f8c-00e9-4c4a-ba78-9054cdd07c5a","Type":"ContainerStarted","Data":"822718bf6ab7cdfa9c22b62cd17a42982533051d86762d075ec88554a94ffeb7"}
Jul 29 17:12:30.149207 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:30.149156    2570 generic.go:356] "Generic (PLEG): container finished" podID="27c541a7-5524-4b02-858e-030f4602f7b8" containerID="6b4aaa460737693d6f42bfb346e82aae2a66750f1cd40c03f75ac49c1aec78a9" exitCode=0
Jul 29 17:12:30.149730 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:30.149232    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" event={"ID":"27c541a7-5524-4b02-858e-030f4602f7b8","Type":"ContainerDied","Data":"6b4aaa460737693d6f42bfb346e82aae2a66750f1cd40c03f75ac49c1aec78a9"}
Jul 29 17:12:31.156745 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.156703    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" event={"ID":"27c541a7-5524-4b02-858e-030f4602f7b8","Type":"ContainerStarted","Data":"9ef5b495f694e5424daff4f0e9a809e28c2b6ebee15c25eef676097fbed65775"}
Jul 29 17:12:31.158724 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.158698    2570 generic.go:356] "Generic (PLEG): container finished" podID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerID="822718bf6ab7cdfa9c22b62cd17a42982533051d86762d075ec88554a94ffeb7" exitCode=0
Jul 29 17:12:31.158864 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.158765    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" event={"ID":"46573f8c-00e9-4c4a-ba78-9054cdd07c5a","Type":"ContainerDied","Data":"822718bf6ab7cdfa9c22b62cd17a42982533051d86762d075ec88554a94ffeb7"}
Jul 29 17:12:31.181215 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.181104    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podStartSLOduration=2.208073512 podStartE2EDuration="7.181080217s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.973006705s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:12:31.178866337 +0000 UTC m=+1279.668386013" watchObservedRunningTime="2026-07-29 17:12:31.181080217 +0000 UTC m=+1279.670599875"
Jul 29 17:12:31.852346 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.852308    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"]
Jul 29 17:12:31.886322 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.886286    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"]
Jul 29 17:12:31.886501 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.886415    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:31.889328 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.889297    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 29 17:12:31.954631 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.954587    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mpkt\" (UniqueName: \"kubernetes.io/projected/a8cf229d-378e-4c83-86a9-b83d3292d0d5-kube-api-access-6mpkt\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:31.954891 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.954873    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cf229d-378e-4c83-86a9-b83d3292d0d5-tls-certs\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:31.955115 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.955100    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-model-cache\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:31.955276 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.955257    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-kserve-provision-location\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:31.955412 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.955396    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-dshm\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:31.955572 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.955557    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-tmp-dir\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:31.955711 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:31.955693    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-home\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.056497 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.056463    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-model-cache\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.056703 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.056510    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-kserve-provision-location\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.056703 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.056532    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-dshm\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.056703 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.056567    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-tmp-dir\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.056703 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.056692    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-home\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.056929 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.056730    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mpkt\" (UniqueName: \"kubernetes.io/projected/a8cf229d-378e-4c83-86a9-b83d3292d0d5-kube-api-access-6mpkt\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.056929 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.056760    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cf229d-378e-4c83-86a9-b83d3292d0d5-tls-certs\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.057039 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.056931    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-model-cache\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.057213 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.057134    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-tmp-dir\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.057213 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.057203    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-home\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.057395 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.057319    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-kserve-provision-location\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.059254 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.059232    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-dshm\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.059444 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.059426    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cf229d-378e-4c83-86a9-b83d3292d0d5-tls-certs\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.065063 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.065026    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mpkt\" (UniqueName: \"kubernetes.io/projected/a8cf229d-378e-4c83-86a9-b83d3292d0d5-kube-api-access-6mpkt\") pod \"router-with-refs-test-kserve-68c8444c64-26gjz\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.169650 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.169552    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" event={"ID":"46573f8c-00e9-4c4a-ba78-9054cdd07c5a","Type":"ContainerStarted","Data":"6b6cf5276b674f59982cd47c51296778a836759192f411c37c9f96e544e4d3b0"}
Jul 29 17:12:32.192784 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.192720    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podStartSLOduration=2.487960821 podStartE2EDuration="8.192699673s" totalImagesPullingTime="811.888993ms" totalInitContainerRuntime="4.892849859s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:12:32.190803936 +0000 UTC m=+1280.680323616" watchObservedRunningTime="2026-07-29 17:12:32.192699673 +0000 UTC m=+1280.682219343"
Jul 29 17:12:32.199340 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.199302    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:32.406805 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:32.406770    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"]
Jul 29 17:12:32.408588 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:12:32.408554    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cf229d_378e_4c83_86a9_b83d3292d0d5.slice/crio-1af68220a228c55cb5ebd813021c9f0f8b3a53712eda9503e7135b3bcc584f5c WatchSource:0}: Error finding container 1af68220a228c55cb5ebd813021c9f0f8b3a53712eda9503e7135b3bcc584f5c: Status 404 returned error can't find the container with id 1af68220a228c55cb5ebd813021c9f0f8b3a53712eda9503e7135b3bcc584f5c
Jul 29 17:12:33.178484 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:33.178440    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" event={"ID":"a8cf229d-378e-4c83-86a9-b83d3292d0d5","Type":"ContainerStarted","Data":"d3a364027920888c4c182be358b7140bde7e894f7f8c213597c9787f5a53c422"}
Jul 29 17:12:33.178944 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:33.178500    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" event={"ID":"a8cf229d-378e-4c83-86a9-b83d3292d0d5","Type":"ContainerStarted","Data":"1af68220a228c55cb5ebd813021c9f0f8b3a53712eda9503e7135b3bcc584f5c"}
Jul 29 17:12:34.730505 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:34.730461    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:34.730936 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:34.730540    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:34.732496 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:34.732461    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:12:34.752180 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:34.752139    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:34.752365 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:34.752235    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:12:34.754050 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:34.754021    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:12:37.953438 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:37.953410    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-fcb85f744-42x5r_688bbea9-c4d1-4004-b979-a71056c37633/main/0.log"
Jul 29 17:12:37.976824 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:37.976787    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:12:38.119271 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.119178    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-tmp-dir\") pod \"688bbea9-c4d1-4004-b979-a71056c37633\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") "
Jul 29 17:12:38.119271 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.119247    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-home\") pod \"688bbea9-c4d1-4004-b979-a71056c37633\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") "
Jul 29 17:12:38.119807 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.119327    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-kserve-provision-location\") pod \"688bbea9-c4d1-4004-b979-a71056c37633\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") "
Jul 29 17:12:38.119807 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.119369    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/688bbea9-c4d1-4004-b979-a71056c37633-tls-certs\") pod \"688bbea9-c4d1-4004-b979-a71056c37633\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") "
Jul 29 17:12:38.119807 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.119402    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgg7j\" (UniqueName: \"kubernetes.io/projected/688bbea9-c4d1-4004-b979-a71056c37633-kube-api-access-cgg7j\") pod \"688bbea9-c4d1-4004-b979-a71056c37633\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") "
Jul 29 17:12:38.119807 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.119436    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-model-cache\") pod \"688bbea9-c4d1-4004-b979-a71056c37633\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") "
Jul 29 17:12:38.119807 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.119472    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-dshm\") pod \"688bbea9-c4d1-4004-b979-a71056c37633\" (UID: \"688bbea9-c4d1-4004-b979-a71056c37633\") "
Jul 29 17:12:38.120150 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.120119    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-home" (OuterVolumeSpecName: "home") pod "688bbea9-c4d1-4004-b979-a71056c37633" (UID: "688bbea9-c4d1-4004-b979-a71056c37633"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:12:38.120866 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.120711    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-model-cache" (OuterVolumeSpecName: "model-cache") pod "688bbea9-c4d1-4004-b979-a71056c37633" (UID: "688bbea9-c4d1-4004-b979-a71056c37633"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:12:38.122774 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.122742    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/688bbea9-c4d1-4004-b979-a71056c37633-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "688bbea9-c4d1-4004-b979-a71056c37633" (UID: "688bbea9-c4d1-4004-b979-a71056c37633"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:12:38.122968 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.122935    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688bbea9-c4d1-4004-b979-a71056c37633-kube-api-access-cgg7j" (OuterVolumeSpecName: "kube-api-access-cgg7j") pod "688bbea9-c4d1-4004-b979-a71056c37633" (UID: "688bbea9-c4d1-4004-b979-a71056c37633"). InnerVolumeSpecName "kube-api-access-cgg7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:12:38.123122 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.123094    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-dshm" (OuterVolumeSpecName: "dshm") pod "688bbea9-c4d1-4004-b979-a71056c37633" (UID: "688bbea9-c4d1-4004-b979-a71056c37633"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:12:38.135738 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.135700    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "688bbea9-c4d1-4004-b979-a71056c37633" (UID: "688bbea9-c4d1-4004-b979-a71056c37633"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:12:38.173921 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.173866    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "688bbea9-c4d1-4004-b979-a71056c37633" (UID: "688bbea9-c4d1-4004-b979-a71056c37633"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:12:38.220483 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.220433    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:12:38.220483 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.220466    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/688bbea9-c4d1-4004-b979-a71056c37633-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:12:38.220483 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.220478    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgg7j\" (UniqueName: \"kubernetes.io/projected/688bbea9-c4d1-4004-b979-a71056c37633-kube-api-access-cgg7j\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:12:38.220483 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.220489    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:12:38.220852 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.220501    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:12:38.220852 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.220509    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:12:38.220852 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.220516    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/688bbea9-c4d1-4004-b979-a71056c37633-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:12:38.233485 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.233447    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-fcb85f744-42x5r_688bbea9-c4d1-4004-b979-a71056c37633/main/0.log"
Jul 29 17:12:38.233914 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.233884    2570 generic.go:356] "Generic (PLEG): container finished" podID="688bbea9-c4d1-4004-b979-a71056c37633" containerID="1a287bb81f905afc89cf8389a862eb8e877f814ef62b8467369a609c1831a4c4" exitCode=137
Jul 29 17:12:38.234050 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.233943    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"
Jul 29 17:12:38.234050 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.233976    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" event={"ID":"688bbea9-c4d1-4004-b979-a71056c37633","Type":"ContainerDied","Data":"1a287bb81f905afc89cf8389a862eb8e877f814ef62b8467369a609c1831a4c4"}
Jul 29 17:12:38.234050 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.234039    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r" event={"ID":"688bbea9-c4d1-4004-b979-a71056c37633","Type":"ContainerDied","Data":"bcec964b12305a072173a4f0f31d55438c56a3fdaadf58ec6c24c91c49fbe59c"}
Jul 29 17:12:38.234245 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.234071    2570 scope.go:117] "RemoveContainer" containerID="1a287bb81f905afc89cf8389a862eb8e877f814ef62b8467369a609c1831a4c4"
Jul 29 17:12:38.236256 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.236229    2570 generic.go:356] "Generic (PLEG): container finished" podID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerID="d3a364027920888c4c182be358b7140bde7e894f7f8c213597c9787f5a53c422" exitCode=0
Jul 29 17:12:38.236387 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.236306    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" event={"ID":"a8cf229d-378e-4c83-86a9-b83d3292d0d5","Type":"ContainerDied","Data":"d3a364027920888c4c182be358b7140bde7e894f7f8c213597c9787f5a53c422"}
Jul 29 17:12:38.245953 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.245932    2570 scope.go:117] "RemoveContainer" containerID="52a13a27a9f0f34f6b9c2af42cf2f9f3b9e62f0366c828413d4055cd5f6cefba"
Jul 29 17:12:38.257710 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.257677    2570 scope.go:117] "RemoveContainer" containerID="1a287bb81f905afc89cf8389a862eb8e877f814ef62b8467369a609c1831a4c4"
Jul 29 17:12:38.258096 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:12:38.258050    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a287bb81f905afc89cf8389a862eb8e877f814ef62b8467369a609c1831a4c4\": container with ID starting with 1a287bb81f905afc89cf8389a862eb8e877f814ef62b8467369a609c1831a4c4 not found: ID does not exist" containerID="1a287bb81f905afc89cf8389a862eb8e877f814ef62b8467369a609c1831a4c4"
Jul 29 17:12:38.258240 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.258089    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a287bb81f905afc89cf8389a862eb8e877f814ef62b8467369a609c1831a4c4"} err="failed to get container status \"1a287bb81f905afc89cf8389a862eb8e877f814ef62b8467369a609c1831a4c4\": rpc error: code = NotFound desc = could not find container \"1a287bb81f905afc89cf8389a862eb8e877f814ef62b8467369a609c1831a4c4\": container with ID starting with 1a287bb81f905afc89cf8389a862eb8e877f814ef62b8467369a609c1831a4c4 not found: ID does not exist"
Jul 29 17:12:38.258240 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.258120    2570 scope.go:117] "RemoveContainer" containerID="52a13a27a9f0f34f6b9c2af42cf2f9f3b9e62f0366c828413d4055cd5f6cefba"
Jul 29 17:12:38.258533 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:12:38.258479    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52a13a27a9f0f34f6b9c2af42cf2f9f3b9e62f0366c828413d4055cd5f6cefba\": container with ID starting with 52a13a27a9f0f34f6b9c2af42cf2f9f3b9e62f0366c828413d4055cd5f6cefba not found: ID does not exist" containerID="52a13a27a9f0f34f6b9c2af42cf2f9f3b9e62f0366c828413d4055cd5f6cefba"
Jul 29 17:12:38.258533 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.258522    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52a13a27a9f0f34f6b9c2af42cf2f9f3b9e62f0366c828413d4055cd5f6cefba"} err="failed to get container status \"52a13a27a9f0f34f6b9c2af42cf2f9f3b9e62f0366c828413d4055cd5f6cefba\": rpc error: code = NotFound desc = could not find container \"52a13a27a9f0f34f6b9c2af42cf2f9f3b9e62f0366c828413d4055cd5f6cefba\": container with ID starting with 52a13a27a9f0f34f6b9c2af42cf2f9f3b9e62f0366c828413d4055cd5f6cefba not found: ID does not exist"
Jul 29 17:12:38.274222 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.274115    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"]
Jul 29 17:12:38.278142 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:38.278107    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-fcb85f744-42x5r"]
Jul 29 17:12:39.249184 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:39.249136    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" event={"ID":"a8cf229d-378e-4c83-86a9-b83d3292d0d5","Type":"ContainerStarted","Data":"35bb773a63b7122f1fdfb2ecc7d6b0074c78e97a23505bec87f5248f33e1eb5f"}
Jul 29 17:12:39.272468 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:39.272412    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podStartSLOduration=3.140159915 podStartE2EDuration="8.272396348s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.132236433s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:12:39.269555832 +0000 UTC m=+1287.759075488" watchObservedRunningTime="2026-07-29 17:12:39.272396348 +0000 UTC m=+1287.761916003"
Jul 29 17:12:39.915370 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:39.915332    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="688bbea9-c4d1-4004-b979-a71056c37633" path="/var/lib/kubelet/pods/688bbea9-c4d1-4004-b979-a71056c37633/volumes"
Jul 29 17:12:42.200540 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:42.200493    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:42.200997 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:42.200573    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:12:42.202513 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:42.202476    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:12:44.730774 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:44.730721    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:12:44.748561 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:44.748528    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:12:44.751873 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:44.751835    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:12:52.199930 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:52.199881    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:12:54.730931 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:54.730869    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:12:54.751937 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:12:54.751889    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:13:02.200220 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:02.200154    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:13:04.730915 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:04.730865    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:13:04.752151 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:04.752103    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:13:12.200327 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:12.200259    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:13:14.731473 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:14.731071    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:13:14.752553 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:14.752459    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:13:22.200408 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:22.200355    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:13:24.730465 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:24.730410    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:13:24.752047 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:24.751998    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:13:32.200688 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:32.200632    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:13:34.731227 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:34.731121    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:13:34.752321 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:34.752278    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:13:42.200322 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:42.200232    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:13:44.730482 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:44.730431    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:13:44.751945 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:44.751901    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:13:52.200350 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:52.200305    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:13:54.730118 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:54.730067    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:13:54.752153 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:13:54.752116    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:14:02.200417 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:02.200373    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:14:04.730342 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:04.730288    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:14:04.751639 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:04.751597    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:14:12.200342 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:12.200294    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:14:14.730778 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:14.730726    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:14:14.751951 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:14.751900    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:14:22.200432 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:22.200388    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:14:24.730409 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:24.730354    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:14:24.751702 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:24.751656    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:14:32.200520 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:32.200473    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:14:34.730974 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:34.730928    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:14:34.752035 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:34.751991    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:14:42.200448 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:42.200400    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:14:44.731092 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:44.731040    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:14:44.752567 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:44.752501    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:14:52.200505 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:52.200449    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:14:54.730241 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:54.730199    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:14:54.752069 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:14:54.752021    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:15:02.200064 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:02.200015    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:15:04.730700 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:04.730599    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:15:04.751734 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:04.751686    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:15:12.199863 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:12.199817    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:15:14.730211 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:14.730142    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:15:14.752073 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:14.752025    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:15:22.200518 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:22.200477    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:15:24.731042 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:24.730987    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:15:24.751804 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:24.751760    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:15:32.200392 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:32.200337    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:15:34.730656 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:34.730602    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 17:15:34.751960 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:34.751911    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 29 17:15:42.200798 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:42.200730    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 29 17:15:44.751743 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:44.751696    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:15:44.754562 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:44.754530    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:15:44.763666 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:44.763629    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:15:44.772132 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:44.772098    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:15:52.210149 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:52.210114    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:15:52.218706 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:15:52.218677    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:16:11.987237 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:16:11.987209    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:16:11.990914 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:16:11.990888    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:21:12.033460 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:21:12.033432    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:21:12.038016 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:21:12.037994    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:26:12.071921 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:26:12.071888    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:26:12.080285 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:26:12.080261    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:27:09.465902 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:09.465859    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"]
Jul 29 17:27:09.466549 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:09.466470    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main" containerID="cri-o://6b6cf5276b674f59982cd47c51296778a836759192f411c37c9f96e544e4d3b0" gracePeriod=30
Jul 29 17:27:09.469756 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:09.469724    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"]
Jul 29 17:27:09.470035 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:09.469984    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main" containerID="cri-o://9ef5b495f694e5424daff4f0e9a809e28c2b6ebee15c25eef676097fbed65775" gracePeriod=30
Jul 29 17:27:16.857635 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.857604    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"]
Jul 29 17:27:16.858227 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.858063    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="storage-initializer"
Jul 29 17:27:16.858227 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.858081    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="storage-initializer"
Jul 29 17:27:16.858227 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.858090    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="main"
Jul 29 17:27:16.858227 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.858098    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="main"
Jul 29 17:27:16.858227 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.858158    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="688bbea9-c4d1-4004-b979-a71056c37633" containerName="main"
Jul 29 17:27:16.861341 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.861323    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:16.863684 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.863661    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\""
Jul 29 17:27:16.882716 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.882682    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"]
Jul 29 17:27:16.935837 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.935788    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:16.935837 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.935836    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:16.936067 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.935910    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:16.936067 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.935952    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj7ds\" (UniqueName: \"kubernetes.io/projected/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-kube-api-access-sj7ds\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:16.936067 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.935977    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:16.936067 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.936027    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:16.936067 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:16.936059    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.036811 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.036767    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.036811 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.036813    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.037055 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.036853    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.037055 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.036879    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7ds\" (UniqueName: \"kubernetes.io/projected/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-kube-api-access-sj7ds\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.037055 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.036900    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.037055 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.036969    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.037055 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.037001    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.037369 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.037264    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.037369 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.037304    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.037481 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.037445    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.037481 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.037470    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.039473 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.039446    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.039681 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.039662    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.046312 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.046286    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7ds\" (UniqueName: \"kubernetes.io/projected/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-kube-api-access-sj7ds\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.172666 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.172564    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:17.338446 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.338258    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"]
Jul 29 17:27:17.341125 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:27:17.341094    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3b1db9_ebdd_4e82_b938_51110ce49ef6.slice/crio-e04f22357c8ec6612cdef6bc772a225d256bf5205a8db0db547b07b02cc2ad92 WatchSource:0}: Error finding container e04f22357c8ec6612cdef6bc772a225d256bf5205a8db0db547b07b02cc2ad92: Status 404 returned error can't find the container with id e04f22357c8ec6612cdef6bc772a225d256bf5205a8db0db547b07b02cc2ad92
Jul 29 17:27:17.463868 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.463765    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" event={"ID":"7e3b1db9-ebdd-4e82-b938-51110ce49ef6","Type":"ContainerStarted","Data":"cb14eff19a05a6bc5ee0e53c1855388de20a919d95a6309d4fe1e95b8d2c0bdc"}
Jul 29 17:27:17.463868 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:17.463826    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" event={"ID":"7e3b1db9-ebdd-4e82-b938-51110ce49ef6","Type":"ContainerStarted","Data":"e04f22357c8ec6612cdef6bc772a225d256bf5205a8db0db547b07b02cc2ad92"}
Jul 29 17:27:18.782930 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:18.782884    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"]
Jul 29 17:27:18.783570 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:18.783240    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main" containerID="cri-o://35bb773a63b7122f1fdfb2ecc7d6b0074c78e97a23505bec87f5248f33e1eb5f" gracePeriod=30
Jul 29 17:27:23.501789 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:23.501754    2570 generic.go:356] "Generic (PLEG): container finished" podID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerID="cb14eff19a05a6bc5ee0e53c1855388de20a919d95a6309d4fe1e95b8d2c0bdc" exitCode=0
Jul 29 17:27:23.502264 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:23.501830    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" event={"ID":"7e3b1db9-ebdd-4e82-b938-51110ce49ef6","Type":"ContainerDied","Data":"cb14eff19a05a6bc5ee0e53c1855388de20a919d95a6309d4fe1e95b8d2c0bdc"}
Jul 29 17:27:24.509890 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:24.509836    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" event={"ID":"7e3b1db9-ebdd-4e82-b938-51110ce49ef6","Type":"ContainerStarted","Data":"8405a66765c4ec8dd5fde47bc8a199ab17e4b2424445fc64f9d3a91e85f6f6c2"}
Jul 29 17:27:24.535680 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:24.535615    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podStartSLOduration=3.123092004 podStartE2EDuration="8.535591926s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.412499922s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:27:24.532874602 +0000 UTC m=+2173.022394262" watchObservedRunningTime="2026-07-29 17:27:24.535591926 +0000 UTC m=+2173.025111583"
Jul 29 17:27:26.080371 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.080338    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"]
Jul 29 17:27:26.084291 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.084267    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.086733 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.086707    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
Jul 29 17:27:26.086932 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.086914    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-5mhh2\""
Jul 29 17:27:26.099971 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.099934    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"]
Jul 29 17:27:26.114078 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.114039    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"]
Jul 29 17:27:26.117741 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.117719    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.135533 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.135500    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"]
Jul 29 17:27:26.219734 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.219693    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.219922 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.219769    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a37f16b1-0a6c-443f-9e67-358b71e01754-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.219922 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.219789    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.219922 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.219809    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.219922 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.219825    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfzdj\" (UniqueName: \"kubernetes.io/projected/a37f16b1-0a6c-443f-9e67-358b71e01754-kube-api-access-xfzdj\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.219922 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.219842    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.219922 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.219860    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npxm8\" (UniqueName: \"kubernetes.io/projected/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-kube-api-access-npxm8\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.219922 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.219913    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.220309 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.219940    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.220309 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.219994    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.220309 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.220033    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.220309 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.220053    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.220309 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.220076    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.220309 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.220121    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.321560 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.321521    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.321759 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.321598    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.321759 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.321632    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.321759 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.321674    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.321759 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.321705    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.321759 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.321737    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.322052 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.321817    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a37f16b1-0a6c-443f-9e67-358b71e01754-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.322052 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.321841    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.322052 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.321872    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.322052 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.321897    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfzdj\" (UniqueName: \"kubernetes.io/projected/a37f16b1-0a6c-443f-9e67-358b71e01754-kube-api-access-xfzdj\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.322052 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.321926    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.322052 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.321955    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npxm8\" (UniqueName: \"kubernetes.io/projected/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-kube-api-access-npxm8\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.322052 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.321983    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.322052 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.322010    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.322052 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.322019    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.322536 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.322390    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.322536 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.322475    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.322656 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.322577    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.322747 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.322718    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.323123 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.322828    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.323123 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.322985    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.323123 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.323082    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.325054 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.325010    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.325420 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.325394    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a37f16b1-0a6c-443f-9e67-358b71e01754-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.325541 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.325519    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.325751 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.325730    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.336301 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.336215    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfzdj\" (UniqueName: \"kubernetes.io/projected/a37f16b1-0a6c-443f-9e67-358b71e01754-kube-api-access-xfzdj\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.337373 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.337318    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npxm8\" (UniqueName: \"kubernetes.io/projected/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-kube-api-access-npxm8\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.394199 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.394147    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:26.432829 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.432793    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:26.577329 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.577291    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"]
Jul 29 17:27:26.579292 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:27:26.579259    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37f16b1_0a6c_443f_9e67_358b71e01754.slice/crio-a9123025bb016ed9e48de29b6e3f6611772659c9a1b5c1bf784b5c79d94b56e3 WatchSource:0}: Error finding container a9123025bb016ed9e48de29b6e3f6611772659c9a1b5c1bf784b5c79d94b56e3: Status 404 returned error can't find the container with id a9123025bb016ed9e48de29b6e3f6611772659c9a1b5c1bf784b5c79d94b56e3
Jul 29 17:27:26.638144 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:26.638115    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"]
Jul 29 17:27:26.640073 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:27:26.640031    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd4b573_6bc0_4a39_b09f_336c1a6cde78.slice/crio-31859851fd615a42cb41d6414a435debb8a41ba5ca08313fd4fe75012734e830 WatchSource:0}: Error finding container 31859851fd615a42cb41d6414a435debb8a41ba5ca08313fd4fe75012734e830: Status 404 returned error can't find the container with id 31859851fd615a42cb41d6414a435debb8a41ba5ca08313fd4fe75012734e830
Jul 29 17:27:27.173413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:27.173366    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:27.173885 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:27.173451    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:27:27.174925 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:27.174897    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:27:27.535069 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:27.535034    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" event={"ID":"a37f16b1-0a6c-443f-9e67-358b71e01754","Type":"ContainerStarted","Data":"47c6f04458e4436b1c21bd1aa28f94c747f5c23eea843aa8903c2e47a7994952"}
Jul 29 17:27:27.535376 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:27.535352    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:27.535518 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:27.535389    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" event={"ID":"a37f16b1-0a6c-443f-9e67-358b71e01754","Type":"ContainerStarted","Data":"a9123025bb016ed9e48de29b6e3f6611772659c9a1b5c1bf784b5c79d94b56e3"}
Jul 29 17:27:27.537848 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:27.537771    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" event={"ID":"bdd4b573-6bc0-4a39-b09f-336c1a6cde78","Type":"ContainerStarted","Data":"41decbc305cce26a52612ea8345b360c1be29f202e85f9d273d69b19d731829e"}
Jul 29 17:27:27.537848 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:27.537835    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" event={"ID":"bdd4b573-6bc0-4a39-b09f-336c1a6cde78","Type":"ContainerStarted","Data":"31859851fd615a42cb41d6414a435debb8a41ba5ca08313fd4fe75012734e830"}
Jul 29 17:27:28.545476 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:28.545424    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" event={"ID":"a37f16b1-0a6c-443f-9e67-358b71e01754","Type":"ContainerStarted","Data":"e285e91129eda121bf926012b378977a78e9e010b57f6c94f0e50b91151c3898"}
Jul 29 17:27:31.569755 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:31.569715    2570 generic.go:356] "Generic (PLEG): container finished" podID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerID="41decbc305cce26a52612ea8345b360c1be29f202e85f9d273d69b19d731829e" exitCode=0
Jul 29 17:27:31.570282 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:31.569796    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" event={"ID":"bdd4b573-6bc0-4a39-b09f-336c1a6cde78","Type":"ContainerDied","Data":"41decbc305cce26a52612ea8345b360c1be29f202e85f9d273d69b19d731829e"}
Jul 29 17:27:32.580555 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:32.580513    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" event={"ID":"bdd4b573-6bc0-4a39-b09f-336c1a6cde78","Type":"ContainerStarted","Data":"320b659e2905fa5592100d693ee2e702911ede9bb0bb2eef9c8f92ac80b72629"}
Jul 29 17:27:32.607428 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:32.605880    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podStartSLOduration=1.921101666 podStartE2EDuration="6.605857233s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.684755567s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:27:32.603497934 +0000 UTC m=+2181.093017589" watchObservedRunningTime="2026-07-29 17:27:32.605857233 +0000 UTC m=+2181.095376892"
Jul 29 17:27:33.589147 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:33.589115    2570 generic.go:356] "Generic (PLEG): container finished" podID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerID="e285e91129eda121bf926012b378977a78e9e010b57f6c94f0e50b91151c3898" exitCode=0
Jul 29 17:27:33.589621 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:33.589191    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" event={"ID":"a37f16b1-0a6c-443f-9e67-358b71e01754","Type":"ContainerDied","Data":"e285e91129eda121bf926012b378977a78e9e010b57f6c94f0e50b91151c3898"}
Jul 29 17:27:34.601520 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:34.601436    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" event={"ID":"a37f16b1-0a6c-443f-9e67-358b71e01754","Type":"ContainerStarted","Data":"6feefb6bcaa30bf263e34abe1243f65fa3c539e3690bb741d94cc10acd4097a5"}
Jul 29 17:27:34.628281 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:34.628217    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podStartSLOduration=3.493692635 podStartE2EDuration="8.628196917s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.134504282s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:27:34.626993134 +0000 UTC m=+2183.116512791" watchObservedRunningTime="2026-07-29 17:27:34.628196917 +0000 UTC m=+2183.117716576"
Jul 29 17:27:36.396658 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:36.396607    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:36.398277 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:36.398246    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:36.398985 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:36.398951    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:27:36.433110 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:36.433057    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:36.433384 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:36.433183    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:27:36.435182 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:36.435124    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:27:36.436824 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:36.436788    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:27:37.173298 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:37.173255    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:27:39.466349 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.466266    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="llm-d-routing-sidecar" containerID="cri-o://c8281c88aa53d5a0e9cef436df858cba8e4284be91c22883bf8d22b10fc1428e" gracePeriod=2
Jul 29 17:27:39.644145 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.644107    2570 generic.go:356] "Generic (PLEG): container finished" podID="27c541a7-5524-4b02-858e-030f4602f7b8" containerID="9ef5b495f694e5424daff4f0e9a809e28c2b6ebee15c25eef676097fbed65775" exitCode=137
Jul 29 17:27:39.644305 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.644191    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" event={"ID":"27c541a7-5524-4b02-858e-030f4602f7b8","Type":"ContainerDied","Data":"9ef5b495f694e5424daff4f0e9a809e28c2b6ebee15c25eef676097fbed65775"}
Jul 29 17:27:39.647050 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.647021    2570 generic.go:356] "Generic (PLEG): container finished" podID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerID="c8281c88aa53d5a0e9cef436df858cba8e4284be91c22883bf8d22b10fc1428e" exitCode=0
Jul 29 17:27:39.647228 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.647097    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" event={"ID":"46573f8c-00e9-4c4a-ba78-9054cdd07c5a","Type":"ContainerDied","Data":"c8281c88aa53d5a0e9cef436df858cba8e4284be91c22883bf8d22b10fc1428e"}
Jul 29 17:27:39.897802 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.897378    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:27:39.957979 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.957954    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-fb85fd64b-9bs98_46573f8c-00e9-4c4a-ba78-9054cdd07c5a/main/0.log"
Jul 29 17:27:39.965781 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.965747    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:27:39.980093 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.980002    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-kserve-provision-location\") pod \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") "
Jul 29 17:27:39.980093 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.980045    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-model-cache\") pod \"27c541a7-5524-4b02-858e-030f4602f7b8\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") "
Jul 29 17:27:39.980093 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.980073    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-home\") pod \"27c541a7-5524-4b02-858e-030f4602f7b8\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") "
Jul 29 17:27:39.980413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.980117    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-model-cache\") pod \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") "
Jul 29 17:27:39.980413 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.980346    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "27c541a7-5524-4b02-858e-030f4602f7b8" (UID: "27c541a7-5524-4b02-858e-030f4602f7b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:39.980583 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.980566    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7qdd\" (UniqueName: \"kubernetes.io/projected/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-kube-api-access-s7qdd\") pod \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") "
Jul 29 17:27:39.980751 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.980736    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-kserve-provision-location\") pod \"27c541a7-5524-4b02-858e-030f4602f7b8\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") "
Jul 29 17:27:39.983657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.980609    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "46573f8c-00e9-4c4a-ba78-9054cdd07c5a" (UID: "46573f8c-00e9-4c4a-ba78-9054cdd07c5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:39.983657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.980875    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c541a7-5524-4b02-858e-030f4602f7b8-tls-certs\") pod \"27c541a7-5524-4b02-858e-030f4602f7b8\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") "
Jul 29 17:27:39.983657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.980906    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-dshm\") pod \"27c541a7-5524-4b02-858e-030f4602f7b8\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") "
Jul 29 17:27:39.983657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.981003    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-tmp-dir\") pod \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") "
Jul 29 17:27:39.983657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.980854    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-home" (OuterVolumeSpecName: "home") pod "27c541a7-5524-4b02-858e-030f4602f7b8" (UID: "27c541a7-5524-4b02-858e-030f4602f7b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:39.983657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.981035    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7sns\" (UniqueName: \"kubernetes.io/projected/27c541a7-5524-4b02-858e-030f4602f7b8-kube-api-access-w7sns\") pod \"27c541a7-5524-4b02-858e-030f4602f7b8\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") "
Jul 29 17:27:39.983657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.981070    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-dshm\") pod \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") "
Jul 29 17:27:39.983657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.981101    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-tmp-dir\") pod \"27c541a7-5524-4b02-858e-030f4602f7b8\" (UID: \"27c541a7-5524-4b02-858e-030f4602f7b8\") "
Jul 29 17:27:39.983657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.981129    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-home\") pod \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") "
Jul 29 17:27:39.983657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.981154    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-tls-certs\") pod \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\" (UID: \"46573f8c-00e9-4c4a-ba78-9054cdd07c5a\") "
Jul 29 17:27:39.983657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.981509    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:39.983657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.981526    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:39.983657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.981539    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:39.998074 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.987777    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-home" (OuterVolumeSpecName: "home") pod "46573f8c-00e9-4c4a-ba78-9054cdd07c5a" (UID: "46573f8c-00e9-4c4a-ba78-9054cdd07c5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:39.998074 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.995757    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-kube-api-access-s7qdd" (OuterVolumeSpecName: "kube-api-access-s7qdd") pod "46573f8c-00e9-4c4a-ba78-9054cdd07c5a" (UID: "46573f8c-00e9-4c4a-ba78-9054cdd07c5a"). InnerVolumeSpecName "kube-api-access-s7qdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:27:39.998074 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.996381    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c541a7-5524-4b02-858e-030f4602f7b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27c541a7-5524-4b02-858e-030f4602f7b8" (UID: "27c541a7-5524-4b02-858e-030f4602f7b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:27:39.998074 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.996572    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-dshm" (OuterVolumeSpecName: "dshm") pod "46573f8c-00e9-4c4a-ba78-9054cdd07c5a" (UID: "46573f8c-00e9-4c4a-ba78-9054cdd07c5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:39.998074 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.996938    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46573f8c-00e9-4c4a-ba78-9054cdd07c5a" (UID: "46573f8c-00e9-4c4a-ba78-9054cdd07c5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:27:39.998710 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:39.998589    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c541a7-5524-4b02-858e-030f4602f7b8-kube-api-access-w7sns" (OuterVolumeSpecName: "kube-api-access-w7sns") pod "27c541a7-5524-4b02-858e-030f4602f7b8" (UID: "27c541a7-5524-4b02-858e-030f4602f7b8"). InnerVolumeSpecName "kube-api-access-w7sns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:27:40.005773 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.005735    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-dshm" (OuterVolumeSpecName: "dshm") pod "27c541a7-5524-4b02-858e-030f4602f7b8" (UID: "27c541a7-5524-4b02-858e-030f4602f7b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:40.009000 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.008969    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "46573f8c-00e9-4c4a-ba78-9054cdd07c5a" (UID: "46573f8c-00e9-4c4a-ba78-9054cdd07c5a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:40.011352 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.011322    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "46573f8c-00e9-4c4a-ba78-9054cdd07c5a" (UID: "46573f8c-00e9-4c4a-ba78-9054cdd07c5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:40.013715 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.013676    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27c541a7-5524-4b02-858e-030f4602f7b8" (UID: "27c541a7-5524-4b02-858e-030f4602f7b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:40.033221 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.033183    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27c541a7-5524-4b02-858e-030f4602f7b8" (UID: "27c541a7-5524-4b02-858e-030f4602f7b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:40.082604 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.082562    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:40.082604 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.082605    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7sns\" (UniqueName: \"kubernetes.io/projected/27c541a7-5524-4b02-858e-030f4602f7b8-kube-api-access-w7sns\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:40.082861 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.082620    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:40.082861 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.082634    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:40.082861 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.082645    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:40.082861 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.082658    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:40.082861 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.082670    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:40.082861 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.082683    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7qdd\" (UniqueName: \"kubernetes.io/projected/46573f8c-00e9-4c4a-ba78-9054cdd07c5a-kube-api-access-s7qdd\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:40.082861 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.082696    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:40.082861 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.082710    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c541a7-5524-4b02-858e-030f4602f7b8-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:40.082861 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.082722    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27c541a7-5524-4b02-858e-030f4602f7b8-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:40.657071 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.657037    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-fb85fd64b-9bs98_46573f8c-00e9-4c4a-ba78-9054cdd07c5a/main/0.log"
Jul 29 17:27:40.657900 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.657874    2570 generic.go:356] "Generic (PLEG): container finished" podID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerID="6b6cf5276b674f59982cd47c51296778a836759192f411c37c9f96e544e4d3b0" exitCode=137
Jul 29 17:27:40.658002 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.657992    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" event={"ID":"46573f8c-00e9-4c4a-ba78-9054cdd07c5a","Type":"ContainerDied","Data":"6b6cf5276b674f59982cd47c51296778a836759192f411c37c9f96e544e4d3b0"}
Jul 29 17:27:40.658072 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.658034    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98" event={"ID":"46573f8c-00e9-4c4a-ba78-9054cdd07c5a","Type":"ContainerDied","Data":"71db65f23118c42f2d134d88cb92b7e44cf3a7da3b1dc5ee6f4c0a2948e1d5f8"}
Jul 29 17:27:40.658072 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.658056    2570 scope.go:117] "RemoveContainer" containerID="6b6cf5276b674f59982cd47c51296778a836759192f411c37c9f96e544e4d3b0"
Jul 29 17:27:40.658210 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.658202    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"
Jul 29 17:27:40.660762 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.660735    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj" event={"ID":"27c541a7-5524-4b02-858e-030f4602f7b8","Type":"ContainerDied","Data":"3ac683b1de95315e12851abd8685fea6fe66cf51260cf031e66e092d9e5cf18b"}
Jul 29 17:27:40.660952 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.660805    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"
Jul 29 17:27:40.671864 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.671825    2570 scope.go:117] "RemoveContainer" containerID="822718bf6ab7cdfa9c22b62cd17a42982533051d86762d075ec88554a94ffeb7"
Jul 29 17:27:40.684989 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.684965    2570 scope.go:117] "RemoveContainer" containerID="c8281c88aa53d5a0e9cef436df858cba8e4284be91c22883bf8d22b10fc1428e"
Jul 29 17:27:40.689394 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.689308    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"]
Jul 29 17:27:40.694525 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.694497    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-58df97877c-zm8tj"]
Jul 29 17:27:40.697881 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.697858    2570 scope.go:117] "RemoveContainer" containerID="6b6cf5276b674f59982cd47c51296778a836759192f411c37c9f96e544e4d3b0"
Jul 29 17:27:40.698259 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:27:40.698234    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6cf5276b674f59982cd47c51296778a836759192f411c37c9f96e544e4d3b0\": container with ID starting with 6b6cf5276b674f59982cd47c51296778a836759192f411c37c9f96e544e4d3b0 not found: ID does not exist" containerID="6b6cf5276b674f59982cd47c51296778a836759192f411c37c9f96e544e4d3b0"
Jul 29 17:27:40.698363 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.698267    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6cf5276b674f59982cd47c51296778a836759192f411c37c9f96e544e4d3b0"} err="failed to get container status \"6b6cf5276b674f59982cd47c51296778a836759192f411c37c9f96e544e4d3b0\": rpc error: code = NotFound desc = could not find container \"6b6cf5276b674f59982cd47c51296778a836759192f411c37c9f96e544e4d3b0\": container with ID starting with 6b6cf5276b674f59982cd47c51296778a836759192f411c37c9f96e544e4d3b0 not found: ID does not exist"
Jul 29 17:27:40.698363 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.698292    2570 scope.go:117] "RemoveContainer" containerID="822718bf6ab7cdfa9c22b62cd17a42982533051d86762d075ec88554a94ffeb7"
Jul 29 17:27:40.698616 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:27:40.698595    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"822718bf6ab7cdfa9c22b62cd17a42982533051d86762d075ec88554a94ffeb7\": container with ID starting with 822718bf6ab7cdfa9c22b62cd17a42982533051d86762d075ec88554a94ffeb7 not found: ID does not exist" containerID="822718bf6ab7cdfa9c22b62cd17a42982533051d86762d075ec88554a94ffeb7"
Jul 29 17:27:40.698706 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.698624    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822718bf6ab7cdfa9c22b62cd17a42982533051d86762d075ec88554a94ffeb7"} err="failed to get container status \"822718bf6ab7cdfa9c22b62cd17a42982533051d86762d075ec88554a94ffeb7\": rpc error: code = NotFound desc = could not find container \"822718bf6ab7cdfa9c22b62cd17a42982533051d86762d075ec88554a94ffeb7\": container with ID starting with 822718bf6ab7cdfa9c22b62cd17a42982533051d86762d075ec88554a94ffeb7 not found: ID does not exist"
Jul 29 17:27:40.698706 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.698645    2570 scope.go:117] "RemoveContainer" containerID="c8281c88aa53d5a0e9cef436df858cba8e4284be91c22883bf8d22b10fc1428e"
Jul 29 17:27:40.698957 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:27:40.698937    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8281c88aa53d5a0e9cef436df858cba8e4284be91c22883bf8d22b10fc1428e\": container with ID starting with c8281c88aa53d5a0e9cef436df858cba8e4284be91c22883bf8d22b10fc1428e not found: ID does not exist" containerID="c8281c88aa53d5a0e9cef436df858cba8e4284be91c22883bf8d22b10fc1428e"
Jul 29 17:27:40.699039 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.698966    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8281c88aa53d5a0e9cef436df858cba8e4284be91c22883bf8d22b10fc1428e"} err="failed to get container status \"c8281c88aa53d5a0e9cef436df858cba8e4284be91c22883bf8d22b10fc1428e\": rpc error: code = NotFound desc = could not find container \"c8281c88aa53d5a0e9cef436df858cba8e4284be91c22883bf8d22b10fc1428e\": container with ID starting with c8281c88aa53d5a0e9cef436df858cba8e4284be91c22883bf8d22b10fc1428e not found: ID does not exist"
Jul 29 17:27:40.699039 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.698988    2570 scope.go:117] "RemoveContainer" containerID="9ef5b495f694e5424daff4f0e9a809e28c2b6ebee15c25eef676097fbed65775"
Jul 29 17:27:40.705705 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.705585    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"]
Jul 29 17:27:40.710141 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.710114    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-fb85fd64b-9bs98"]
Jul 29 17:27:40.712557 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:40.712537    2570 scope.go:117] "RemoveContainer" containerID="6b4aaa460737693d6f42bfb346e82aae2a66750f1cd40c03f75ac49c1aec78a9"
Jul 29 17:27:41.916741 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:41.916707    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" path="/var/lib/kubelet/pods/27c541a7-5524-4b02-858e-030f4602f7b8/volumes"
Jul 29 17:27:41.917265 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:41.917139    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" path="/var/lib/kubelet/pods/46573f8c-00e9-4c4a-ba78-9054cdd07c5a/volumes"
Jul 29 17:27:46.394962 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:46.394913    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:27:46.433241 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:46.433149    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:27:47.173963 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:47.173909    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:27:49.235486 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.235456    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-68c8444c64-26gjz_a8cf229d-378e-4c83-86a9-b83d3292d0d5/main/0.log"
Jul 29 17:27:49.268317 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.268273    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:27:49.378459 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.378357    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-model-cache\") pod \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") "
Jul 29 17:27:49.378459 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.378398    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-tmp-dir\") pod \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") "
Jul 29 17:27:49.378459 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.378432    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-kserve-provision-location\") pod \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") "
Jul 29 17:27:49.378747 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.378469    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-dshm\") pod \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") "
Jul 29 17:27:49.378747 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.378489    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-home\") pod \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") "
Jul 29 17:27:49.378747 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.378554    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mpkt\" (UniqueName: \"kubernetes.io/projected/a8cf229d-378e-4c83-86a9-b83d3292d0d5-kube-api-access-6mpkt\") pod \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") "
Jul 29 17:27:49.378747 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.378600    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cf229d-378e-4c83-86a9-b83d3292d0d5-tls-certs\") pod \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\" (UID: \"a8cf229d-378e-4c83-86a9-b83d3292d0d5\") "
Jul 29 17:27:49.379156 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.379100    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-home" (OuterVolumeSpecName: "home") pod "a8cf229d-378e-4c83-86a9-b83d3292d0d5" (UID: "a8cf229d-378e-4c83-86a9-b83d3292d0d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:49.381846 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.381799    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cf229d-378e-4c83-86a9-b83d3292d0d5-kube-api-access-6mpkt" (OuterVolumeSpecName: "kube-api-access-6mpkt") pod "a8cf229d-378e-4c83-86a9-b83d3292d0d5" (UID: "a8cf229d-378e-4c83-86a9-b83d3292d0d5"). InnerVolumeSpecName "kube-api-access-6mpkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:27:49.382065 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.382037    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8cf229d-378e-4c83-86a9-b83d3292d0d5" (UID: "a8cf229d-378e-4c83-86a9-b83d3292d0d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:49.382694 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.382629    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-dshm" (OuterVolumeSpecName: "dshm") pod "a8cf229d-378e-4c83-86a9-b83d3292d0d5" (UID: "a8cf229d-378e-4c83-86a9-b83d3292d0d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:49.382694 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.382661    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cf229d-378e-4c83-86a9-b83d3292d0d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8cf229d-378e-4c83-86a9-b83d3292d0d5" (UID: "a8cf229d-378e-4c83-86a9-b83d3292d0d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:27:49.400367 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.400327    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8cf229d-378e-4c83-86a9-b83d3292d0d5" (UID: "a8cf229d-378e-4c83-86a9-b83d3292d0d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:49.407003 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.406969    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8cf229d-378e-4c83-86a9-b83d3292d0d5" (UID: "a8cf229d-378e-4c83-86a9-b83d3292d0d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:49.479527 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.479493    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mpkt\" (UniqueName: \"kubernetes.io/projected/a8cf229d-378e-4c83-86a9-b83d3292d0d5-kube-api-access-6mpkt\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:49.479527 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.479516    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cf229d-378e-4c83-86a9-b83d3292d0d5-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:49.479527 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.479529    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:49.479784 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.479540    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:49.479784 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.479552    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:49.479784 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.479563    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:49.479784 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.479574    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cf229d-378e-4c83-86a9-b83d3292d0d5-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:27:49.730763 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.730680    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-68c8444c64-26gjz_a8cf229d-378e-4c83-86a9-b83d3292d0d5/main/0.log"
Jul 29 17:27:49.731136 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.731111    2570 generic.go:356] "Generic (PLEG): container finished" podID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerID="35bb773a63b7122f1fdfb2ecc7d6b0074c78e97a23505bec87f5248f33e1eb5f" exitCode=137
Jul 29 17:27:49.731273 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.731199    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" event={"ID":"a8cf229d-378e-4c83-86a9-b83d3292d0d5","Type":"ContainerDied","Data":"35bb773a63b7122f1fdfb2ecc7d6b0074c78e97a23505bec87f5248f33e1eb5f"}
Jul 29 17:27:49.731273 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.731224    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"
Jul 29 17:27:49.731407 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.731318    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz" event={"ID":"a8cf229d-378e-4c83-86a9-b83d3292d0d5","Type":"ContainerDied","Data":"1af68220a228c55cb5ebd813021c9f0f8b3a53712eda9503e7135b3bcc584f5c"}
Jul 29 17:27:49.731407 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.731354    2570 scope.go:117] "RemoveContainer" containerID="35bb773a63b7122f1fdfb2ecc7d6b0074c78e97a23505bec87f5248f33e1eb5f"
Jul 29 17:27:49.744244 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.744225    2570 scope.go:117] "RemoveContainer" containerID="d3a364027920888c4c182be358b7140bde7e894f7f8c213597c9787f5a53c422"
Jul 29 17:27:49.762616 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.762588    2570 scope.go:117] "RemoveContainer" containerID="35bb773a63b7122f1fdfb2ecc7d6b0074c78e97a23505bec87f5248f33e1eb5f"
Jul 29 17:27:49.763488 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:27:49.763453    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35bb773a63b7122f1fdfb2ecc7d6b0074c78e97a23505bec87f5248f33e1eb5f\": container with ID starting with 35bb773a63b7122f1fdfb2ecc7d6b0074c78e97a23505bec87f5248f33e1eb5f not found: ID does not exist" containerID="35bb773a63b7122f1fdfb2ecc7d6b0074c78e97a23505bec87f5248f33e1eb5f"
Jul 29 17:27:49.763630 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.763497    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35bb773a63b7122f1fdfb2ecc7d6b0074c78e97a23505bec87f5248f33e1eb5f"} err="failed to get container status \"35bb773a63b7122f1fdfb2ecc7d6b0074c78e97a23505bec87f5248f33e1eb5f\": rpc error: code = NotFound desc = could not find container \"35bb773a63b7122f1fdfb2ecc7d6b0074c78e97a23505bec87f5248f33e1eb5f\": container with ID starting with 35bb773a63b7122f1fdfb2ecc7d6b0074c78e97a23505bec87f5248f33e1eb5f not found: ID does not exist"
Jul 29 17:27:49.763630 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.763526    2570 scope.go:117] "RemoveContainer" containerID="d3a364027920888c4c182be358b7140bde7e894f7f8c213597c9787f5a53c422"
Jul 29 17:27:49.764115 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:27:49.764093    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3a364027920888c4c182be358b7140bde7e894f7f8c213597c9787f5a53c422\": container with ID starting with d3a364027920888c4c182be358b7140bde7e894f7f8c213597c9787f5a53c422 not found: ID does not exist" containerID="d3a364027920888c4c182be358b7140bde7e894f7f8c213597c9787f5a53c422"
Jul 29 17:27:49.764308 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.764281    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a364027920888c4c182be358b7140bde7e894f7f8c213597c9787f5a53c422"} err="failed to get container status \"d3a364027920888c4c182be358b7140bde7e894f7f8c213597c9787f5a53c422\": rpc error: code = NotFound desc = could not find container \"d3a364027920888c4c182be358b7140bde7e894f7f8c213597c9787f5a53c422\": container with ID starting with d3a364027920888c4c182be358b7140bde7e894f7f8c213597c9787f5a53c422 not found: ID does not exist"
Jul 29 17:27:49.765065 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.765036    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"]
Jul 29 17:27:49.768525 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.768500    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-68c8444c64-26gjz"]
Jul 29 17:27:49.917980 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:49.917939    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" path="/var/lib/kubelet/pods/a8cf229d-378e-4c83-86a9-b83d3292d0d5/volumes"
Jul 29 17:27:56.395494 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:56.395447    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:27:56.434008 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:56.433961    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:27:57.173734 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:27:57.173683    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:28:06.395074 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:06.395023    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:28:06.433835 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:06.433773    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:28:07.173964 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:07.173914    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:28:16.394973 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:16.394907    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:28:16.433921 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:16.433877    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:28:17.174010 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:17.173950    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:28:26.394823 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:26.394771    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:28:26.433595 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:26.433535    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:28:27.173915 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:27.173836    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:28:36.395318 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:36.395208    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:28:36.434069 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:36.434020    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:28:37.173637 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:37.173591    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:28:46.394760 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:46.394703    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:28:46.433386 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:46.433335    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:28:47.173593 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:47.173537    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:28:56.394770 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:56.394712    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:28:56.433560 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:56.433510    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:28:57.173371 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:28:57.173324    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:29:06.395029 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:06.394979    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:29:06.433315 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:06.433270    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:29:07.173704 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:07.173658    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:29:16.394826 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:16.394774    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:29:16.433980 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:16.433935    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:29:17.173597 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:17.173545    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:29:26.395106 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:26.395056    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:29:26.433374 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:26.433318    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:29:27.173347 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:27.173297    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:29:36.395027 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:36.394976    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:29:36.433346 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:36.433296    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:29:37.173655 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:37.173600    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:29:46.395248 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:46.395203    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:29:46.433246 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:46.433199    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:29:47.173799 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:47.173745    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:29:56.394895 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:56.394848    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:29:56.434235 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:56.434195    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:29:57.174060 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:29:57.174014    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:30:06.394716 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:06.394590    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:30:06.433214 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:06.433155    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:30:07.173854 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:07.173799    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:30:16.395156 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:16.395094    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:30:16.433427 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:16.433385    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:30:17.173229 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:17.173187    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 29 17:30:26.394696 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:26.394648    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:30:26.433519 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:26.433470    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:30:27.183126 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:27.183089    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:30:27.191788 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:27.191760    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:30:36.395233 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:36.395158    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Jul 29 17:30:36.433429 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:36.433379    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 17:30:38.840287 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:38.840240    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"]
Jul 29 17:30:38.840807 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:38.840489    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main" containerID="cri-o://8405a66765c4ec8dd5fde47bc8a199ab17e4b2424445fc64f9d3a91e85f6f6c2" gracePeriod=30
Jul 29 17:30:46.412041 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:46.412007    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:30:46.427947 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:46.427910    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:30:46.445048 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:46.445008    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:30:46.453367 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:46.453334    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:30:54.747020 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.746983    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 29 17:30:54.747578 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747444    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main"
Jul 29 17:30:54.747578 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747457    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main"
Jul 29 17:30:54.747578 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747473    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="storage-initializer"
Jul 29 17:30:54.747578 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747479    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="storage-initializer"
Jul 29 17:30:54.747578 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747486    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main"
Jul 29 17:30:54.747578 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747490    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main"
Jul 29 17:30:54.747578 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747504    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="storage-initializer"
Jul 29 17:30:54.747578 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747511    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="storage-initializer"
Jul 29 17:30:54.747578 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747518    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="storage-initializer"
Jul 29 17:30:54.747578 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747523    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="storage-initializer"
Jul 29 17:30:54.747578 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747529    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main"
Jul 29 17:30:54.747578 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747534    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main"
Jul 29 17:30:54.747578 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747542    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="llm-d-routing-sidecar"
Jul 29 17:30:54.747578 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747549    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="llm-d-routing-sidecar"
Jul 29 17:30:54.748065 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747609    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="main"
Jul 29 17:30:54.748065 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747618    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="46573f8c-00e9-4c4a-ba78-9054cdd07c5a" containerName="llm-d-routing-sidecar"
Jul 29 17:30:54.748065 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747625    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8cf229d-378e-4c83-86a9-b83d3292d0d5" containerName="main"
Jul 29 17:30:54.748065 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.747630    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c541a7-5524-4b02-858e-030f4602f7b8" containerName="main"
Jul 29 17:30:54.750840 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.750820    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.754100 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.754076    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-tj6gb\""
Jul 29 17:30:54.754260 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.754219    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
Jul 29 17:30:54.766565 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.766539    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 29 17:30:54.875404 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.875352    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pdlk\" (UniqueName: \"kubernetes.io/projected/ba0c65d5-ed23-442e-9779-076f89fd7280-kube-api-access-9pdlk\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.875404 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.875408    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.875667 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.875432    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.875667 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.875471    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.875667 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.875488    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba0c65d5-ed23-442e-9779-076f89fd7280-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.875667 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.875553    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.875667 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.875597    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.977086 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.977049    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdlk\" (UniqueName: \"kubernetes.io/projected/ba0c65d5-ed23-442e-9779-076f89fd7280-kube-api-access-9pdlk\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.977304 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.977095    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.977304 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.977114    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.977304 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.977192    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.977304 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.977216    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba0c65d5-ed23-442e-9779-076f89fd7280-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.977304 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.977258    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.977304 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.977285    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.977644 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.977583    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.977712 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.977640    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.977712 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.977659    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.977712 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.977702    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.979537 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.979511    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.979680 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.979653    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba0c65d5-ed23-442e-9779-076f89fd7280-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:54.985657 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:54.985627    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pdlk\" (UniqueName: \"kubernetes.io/projected/ba0c65d5-ed23-442e-9779-076f89fd7280-kube-api-access-9pdlk\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:55.064683 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:55.064646    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:30:55.219709 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:55.219681    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 29 17:30:55.221625 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:30:55.221586    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0c65d5_ed23_442e_9779_076f89fd7280.slice/crio-6eeb5c4dc4534480fb388618184a41e9a8837b28bde597758ca137a03dfabbd1 WatchSource:0}: Error finding container 6eeb5c4dc4534480fb388618184a41e9a8837b28bde597758ca137a03dfabbd1: Status 404 returned error can't find the container with id 6eeb5c4dc4534480fb388618184a41e9a8837b28bde597758ca137a03dfabbd1
Jul 29 17:30:55.899533 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:55.899495    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"ba0c65d5-ed23-442e-9779-076f89fd7280","Type":"ContainerStarted","Data":"9bcbc19ce3e21c1799f84600d28771270ca9ba85bf3589ffadf22af113b6284a"}
Jul 29 17:30:55.900021 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:30:55.899548    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"ba0c65d5-ed23-442e-9779-076f89fd7280","Type":"ContainerStarted","Data":"6eeb5c4dc4534480fb388618184a41e9a8837b28bde597758ca137a03dfabbd1"}
Jul 29 17:31:00.938564 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:00.938522    2570 generic.go:356] "Generic (PLEG): container finished" podID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerID="9bcbc19ce3e21c1799f84600d28771270ca9ba85bf3589ffadf22af113b6284a" exitCode=0
Jul 29 17:31:00.938958 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:00.938588    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"ba0c65d5-ed23-442e-9779-076f89fd7280","Type":"ContainerDied","Data":"9bcbc19ce3e21c1799f84600d28771270ca9ba85bf3589ffadf22af113b6284a"}
Jul 29 17:31:01.947226 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:01.947188    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"ba0c65d5-ed23-442e-9779-076f89fd7280","Type":"ContainerStarted","Data":"62d2c12a2208a37fe04bd46837d8333ca3e708c5c06bfb96263437881c046cc7"}
Jul 29 17:31:01.968805 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:01.968741    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=3.146957061 podStartE2EDuration="7.968717619s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.821760558s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:31:01.967561376 +0000 UTC m=+2390.457081056" watchObservedRunningTime="2026-07-29 17:31:01.968717619 +0000 UTC m=+2390.458237278"
Jul 29 17:31:05.065069 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:05.065024    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:31:05.066930 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:05.066898    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 17:31:08.995778 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:08.995751    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l_7e3b1db9-ebdd-4e82-b938-51110ce49ef6/main/0.log"
Jul 29 17:31:08.996201 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:08.996137    2570 generic.go:356] "Generic (PLEG): container finished" podID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerID="8405a66765c4ec8dd5fde47bc8a199ab17e4b2424445fc64f9d3a91e85f6f6c2" exitCode=137
Jul 29 17:31:08.996201 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:08.996190    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" event={"ID":"7e3b1db9-ebdd-4e82-b938-51110ce49ef6","Type":"ContainerDied","Data":"8405a66765c4ec8dd5fde47bc8a199ab17e4b2424445fc64f9d3a91e85f6f6c2"}
Jul 29 17:31:09.214939 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.214890    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l_7e3b1db9-ebdd-4e82-b938-51110ce49ef6/main/0.log"
Jul 29 17:31:09.236022 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.235502    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:31:09.321680 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.321619    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-tmp-dir\") pod \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") "
Jul 29 17:31:09.321680 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.321681    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-home\") pod \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") "
Jul 29 17:31:09.321941 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.321746    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-kserve-provision-location\") pod \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") "
Jul 29 17:31:09.321941 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.321791    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj7ds\" (UniqueName: \"kubernetes.io/projected/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-kube-api-access-sj7ds\") pod \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") "
Jul 29 17:31:09.321941 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.321841    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-model-cache\") pod \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") "
Jul 29 17:31:09.321941 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.321919    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-dshm\") pod \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") "
Jul 29 17:31:09.322150 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.321947    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-tls-certs\") pod \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\" (UID: \"7e3b1db9-ebdd-4e82-b938-51110ce49ef6\") "
Jul 29 17:31:09.322355 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.322326    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e3b1db9-ebdd-4e82-b938-51110ce49ef6" (UID: "7e3b1db9-ebdd-4e82-b938-51110ce49ef6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:31:09.323012 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.322971    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-home" (OuterVolumeSpecName: "home") pod "7e3b1db9-ebdd-4e82-b938-51110ce49ef6" (UID: "7e3b1db9-ebdd-4e82-b938-51110ce49ef6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:31:09.341407 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.341335    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-dshm" (OuterVolumeSpecName: "dshm") pod "7e3b1db9-ebdd-4e82-b938-51110ce49ef6" (UID: "7e3b1db9-ebdd-4e82-b938-51110ce49ef6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:31:09.342316 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.342278    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e3b1db9-ebdd-4e82-b938-51110ce49ef6" (UID: "7e3b1db9-ebdd-4e82-b938-51110ce49ef6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:31:09.344221 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.342663    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-kube-api-access-sj7ds" (OuterVolumeSpecName: "kube-api-access-sj7ds") pod "7e3b1db9-ebdd-4e82-b938-51110ce49ef6" (UID: "7e3b1db9-ebdd-4e82-b938-51110ce49ef6"). InnerVolumeSpecName "kube-api-access-sj7ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:31:09.344221 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.343714    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e3b1db9-ebdd-4e82-b938-51110ce49ef6" (UID: "7e3b1db9-ebdd-4e82-b938-51110ce49ef6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:31:09.395140 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.395098    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e3b1db9-ebdd-4e82-b938-51110ce49ef6" (UID: "7e3b1db9-ebdd-4e82-b938-51110ce49ef6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:31:09.423398 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.423346    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj7ds\" (UniqueName: \"kubernetes.io/projected/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-kube-api-access-sj7ds\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:31:09.423398 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.423390    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:31:09.423398 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.423404    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:31:09.423715 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.423416    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:31:09.423715 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.423428    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:31:09.423715 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.423439    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:31:09.423715 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:09.423451    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e3b1db9-ebdd-4e82-b938-51110ce49ef6-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:31:10.004061 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:10.004025    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l_7e3b1db9-ebdd-4e82-b938-51110ce49ef6/main/0.log"
Jul 29 17:31:10.005565 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:10.005305    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l" event={"ID":"7e3b1db9-ebdd-4e82-b938-51110ce49ef6","Type":"ContainerDied","Data":"e04f22357c8ec6612cdef6bc772a225d256bf5205a8db0db547b07b02cc2ad92"}
Jul 29 17:31:10.005565 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:10.005359    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"
Jul 29 17:31:10.005565 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:10.005403    2570 scope.go:117] "RemoveContainer" containerID="8405a66765c4ec8dd5fde47bc8a199ab17e4b2424445fc64f9d3a91e85f6f6c2"
Jul 29 17:31:10.023560 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:10.023531    2570 scope.go:117] "RemoveContainer" containerID="cb14eff19a05a6bc5ee0e53c1855388de20a919d95a6309d4fe1e95b8d2c0bdc"
Jul 29 17:31:10.040650 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:10.037761    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"]
Jul 29 17:31:10.044492 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:10.044439    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-79b6995bcbrb97l"]
Jul 29 17:31:11.915813 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:11.915772    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" path="/var/lib/kubelet/pods/7e3b1db9-ebdd-4e82-b938-51110ce49ef6/volumes"
Jul 29 17:31:12.115845 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:12.115725    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:31:12.452935 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:12.125628    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:31:15.065580 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:15.065530    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 17:31:25.064884 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:25.064847    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:31:25.065335 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:25.065143    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 17:31:29.507663 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.507600    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"]
Jul 29 17:31:29.508927 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.508255    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="storage-initializer"
Jul 29 17:31:29.508927 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.508284    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="storage-initializer"
Jul 29 17:31:29.508927 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.508305    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main"
Jul 29 17:31:29.508927 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.508313    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main"
Jul 29 17:31:29.508927 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.508471    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3b1db9-ebdd-4e82-b938-51110ce49ef6" containerName="main"
Jul 29 17:31:29.512897 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.512863    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.515536 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.515467    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-klfmr\""
Jul 29 17:31:29.515771 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.515467    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 29 17:31:29.521997 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.521468    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"]
Jul 29 17:31:29.528408 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.528374    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"]
Jul 29 17:31:29.528598 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.528532    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.534627 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.534548    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"]
Jul 29 17:31:29.636632 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.636597    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-model-cache\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.636632 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.636635    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-dshm\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.636847 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.636661    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.636847 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.636692    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-home\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.636847 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.636741    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.636847 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.636788    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f55dce6-b3a0-4631-ba6a-a77936e41175-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.636847 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.636805    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.636847 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.636838    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.637050 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.636859    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.637050 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.636888    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7sfz\" (UniqueName: \"kubernetes.io/projected/1f55dce6-b3a0-4631-ba6a-a77936e41175-kube-api-access-k7sfz\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.637050 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.636904    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.637050 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.636921    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k92z\" (UniqueName: \"kubernetes.io/projected/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-kube-api-access-2k92z\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.637050 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.636988    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.637050 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.637010    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.737974 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.737939    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-model-cache\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.737974 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.737977    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-dshm\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.738322 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.738005    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.738322 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.738031    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-home\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.738322 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.738049    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.738322 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.738077    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f55dce6-b3a0-4631-ba6a-a77936e41175-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.738539 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.738322    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.738539 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.738390    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.738539 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.738421    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.738539 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.738446    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7sfz\" (UniqueName: \"kubernetes.io/projected/1f55dce6-b3a0-4631-ba6a-a77936e41175-kube-api-access-k7sfz\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.738539 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.738479    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.738539 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.738484    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-model-cache\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.738539 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.738509    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.739798 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.739188    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.739798 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.739293    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.739798 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.739451    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.739798 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.739609    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.740100 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.739948    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-home\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.742739 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.742712    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f55dce6-b3a0-4631-ba6a-a77936e41175-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.742856 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.742748    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.746369 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.746344    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k92z\" (UniqueName: \"kubernetes.io/projected/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-kube-api-access-2k92z\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.746602 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.746584    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.746722 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.746707    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.747144 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.747123    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.750226 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.750181    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k92z\" (UniqueName: \"kubernetes.io/projected/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-kube-api-access-2k92z\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.750505 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.750473    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-dshm\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.751844 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.751813    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:29.751844 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.751811    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7sfz\" (UniqueName: \"kubernetes.io/projected/1f55dce6-b3a0-4631-ba6a-a77936e41175-kube-api-access-k7sfz\") pod \"custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.830443 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.830338    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:29.845232 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:29.845193    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:30.021536 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:30.021489    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"]
Jul 29 17:31:30.021808 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:31:30.021772    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f55dce6_b3a0_4631_ba6a_a77936e41175.slice/crio-943663e1155980fcca090f11efa3c2de1b229e3948a4af2cd6e8be0d0a453dee WatchSource:0}: Error finding container 943663e1155980fcca090f11efa3c2de1b229e3948a4af2cd6e8be0d0a453dee: Status 404 returned error can't find the container with id 943663e1155980fcca090f11efa3c2de1b229e3948a4af2cd6e8be0d0a453dee
Jul 29 17:31:30.055996 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:30.055968    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"]
Jul 29 17:31:30.057849 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:31:30.057767    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ee2bb9_7aca_4e66_96e0_d8bcaa6f557c.slice/crio-3b920dd80da2b469ba2f98a283efe4d4024da57aef23044862ad5f6c1f0d3b0a WatchSource:0}: Error finding container 3b920dd80da2b469ba2f98a283efe4d4024da57aef23044862ad5f6c1f0d3b0a: Status 404 returned error can't find the container with id 3b920dd80da2b469ba2f98a283efe4d4024da57aef23044862ad5f6c1f0d3b0a
Jul 29 17:31:30.148327 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:30.148227    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" event={"ID":"1f55dce6-b3a0-4631-ba6a-a77936e41175","Type":"ContainerStarted","Data":"792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197"}
Jul 29 17:31:30.148327 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:30.148302    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" event={"ID":"1f55dce6-b3a0-4631-ba6a-a77936e41175","Type":"ContainerStarted","Data":"943663e1155980fcca090f11efa3c2de1b229e3948a4af2cd6e8be0d0a453dee"}
Jul 29 17:31:30.148594 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:30.148389    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:30.150224 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:30.150196    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" event={"ID":"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c","Type":"ContainerStarted","Data":"3c16782601b0ce9e3188e236854b5865e76115d04dab6811db3502b6591eecdf"}
Jul 29 17:31:30.150351 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:30.150251    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" event={"ID":"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c","Type":"ContainerStarted","Data":"3b920dd80da2b469ba2f98a283efe4d4024da57aef23044862ad5f6c1f0d3b0a"}
Jul 29 17:31:31.163939 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:31.163895    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" event={"ID":"1f55dce6-b3a0-4631-ba6a-a77936e41175","Type":"ContainerStarted","Data":"314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398"}
Jul 29 17:31:32.284811 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:32.284771    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"]
Jul 29 17:31:32.285686 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:32.285648    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main" containerID="cri-o://320b659e2905fa5592100d693ee2e702911ede9bb0bb2eef9c8f92ac80b72629" gracePeriod=30
Jul 29 17:31:32.289855 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:32.289823    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"]
Jul 29 17:31:32.290382 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:32.290343    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main" containerID="cri-o://6feefb6bcaa30bf263e34abe1243f65fa3c539e3690bb741d94cc10acd4097a5" gracePeriod=30
Jul 29 17:31:35.065574 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:35.065452    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 17:31:36.202545 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:36.202463    2570 generic.go:356] "Generic (PLEG): container finished" podID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerID="314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398" exitCode=0
Jul 29 17:31:36.202545 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:36.202527    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" event={"ID":"1f55dce6-b3a0-4631-ba6a-a77936e41175","Type":"ContainerDied","Data":"314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398"}
Jul 29 17:31:36.204456 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:36.204425    2570 generic.go:356] "Generic (PLEG): container finished" podID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerID="3c16782601b0ce9e3188e236854b5865e76115d04dab6811db3502b6591eecdf" exitCode=0
Jul 29 17:31:36.204597 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:36.204482    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" event={"ID":"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c","Type":"ContainerDied","Data":"3c16782601b0ce9e3188e236854b5865e76115d04dab6811db3502b6591eecdf"}
Jul 29 17:31:37.217140 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:37.217099    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" event={"ID":"1f55dce6-b3a0-4631-ba6a-a77936e41175","Type":"ContainerStarted","Data":"0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c"}
Jul 29 17:31:37.219507 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:37.219469    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" event={"ID":"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c","Type":"ContainerStarted","Data":"3b121309b807cea8f9d52067430791648b76e5fcc4d36fb9d630a5cc51f7f70e"}
Jul 29 17:31:37.245629 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:37.245418    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podStartSLOduration=2.529207463 podStartE2EDuration="8.245393218s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.716185755s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:31:37.240701 +0000 UTC m=+2425.730220658" watchObservedRunningTime="2026-07-29 17:31:37.245393218 +0000 UTC m=+2425.734912875"
Jul 29 17:31:37.262487 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:37.262414    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podStartSLOduration=2.440918939 podStartE2EDuration="8.262390774s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.821471835s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:31:37.259702476 +0000 UTC m=+2425.749222149" watchObservedRunningTime="2026-07-29 17:31:37.262390774 +0000 UTC m=+2425.751910430"
Jul 29 17:31:39.830940 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:39.830902    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:39.831485 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:39.831005    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:39.833017 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:39.832980    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:31:39.846375 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:39.846322    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:39.846567 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:39.846409    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:31:39.848290 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:39.848230    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:31:45.065983 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:45.065933    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 17:31:49.831825 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:49.831779    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:31:49.846087 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:49.846040    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:31:49.849541 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:49.849502    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:31:55.066028 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:55.065975    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 17:31:59.831243 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:59.831194    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:31:59.845922 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:31:59.845883    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:32:02.291122 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.291032    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="llm-d-routing-sidecar" containerID="cri-o://47c6f04458e4436b1c21bd1aa28f94c747f5c23eea843aa8903c2e47a7994952" gracePeriod=2
Jul 29 17:32:02.416514 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.416470    2570 generic.go:356] "Generic (PLEG): container finished" podID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerID="47c6f04458e4436b1c21bd1aa28f94c747f5c23eea843aa8903c2e47a7994952" exitCode=0
Jul 29 17:32:02.416699 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.416543    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" event={"ID":"a37f16b1-0a6c-443f-9e67-358b71e01754","Type":"ContainerDied","Data":"47c6f04458e4436b1c21bd1aa28f94c747f5c23eea843aa8903c2e47a7994952"}
Jul 29 17:32:02.762292 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.762259    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:32:02.768027 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.767997    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn_a37f16b1-0a6c-443f-9e67-358b71e01754/main/0.log"
Jul 29 17:32:02.778943 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.778905    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:32:02.901826 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.901721    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-tmp-dir\") pod \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") "
Jul 29 17:32:02.901826 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.901826    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-dshm\") pod \"a37f16b1-0a6c-443f-9e67-358b71e01754\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") "
Jul 29 17:32:02.902083 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.901874    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-home\") pod \"a37f16b1-0a6c-443f-9e67-358b71e01754\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") "
Jul 29 17:32:02.902083 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.901915    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-model-cache\") pod \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") "
Jul 29 17:32:02.902083 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.901941    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-dshm\") pod \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") "
Jul 29 17:32:02.902083 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.901974    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-kserve-provision-location\") pod \"a37f16b1-0a6c-443f-9e67-358b71e01754\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") "
Jul 29 17:32:02.902083 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.902000    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a37f16b1-0a6c-443f-9e67-358b71e01754-tls-certs\") pod \"a37f16b1-0a6c-443f-9e67-358b71e01754\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") "
Jul 29 17:32:02.902083 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.902041    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfzdj\" (UniqueName: \"kubernetes.io/projected/a37f16b1-0a6c-443f-9e67-358b71e01754-kube-api-access-xfzdj\") pod \"a37f16b1-0a6c-443f-9e67-358b71e01754\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") "
Jul 29 17:32:02.902083 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.902072    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-home\") pod \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") "
Jul 29 17:32:02.902463 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.902115    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-kserve-provision-location\") pod \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") "
Jul 29 17:32:02.902463 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.902182    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-tls-certs\") pod \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") "
Jul 29 17:32:02.902463 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.902213    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-tmp-dir\") pod \"a37f16b1-0a6c-443f-9e67-358b71e01754\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") "
Jul 29 17:32:02.902463 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.902238    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npxm8\" (UniqueName: \"kubernetes.io/projected/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-kube-api-access-npxm8\") pod \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\" (UID: \"bdd4b573-6bc0-4a39-b09f-336c1a6cde78\") "
Jul 29 17:32:02.902463 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.902279    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-model-cache\") pod \"a37f16b1-0a6c-443f-9e67-358b71e01754\" (UID: \"a37f16b1-0a6c-443f-9e67-358b71e01754\") "
Jul 29 17:32:02.903060 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.902825    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-model-cache" (OuterVolumeSpecName: "model-cache") pod "a37f16b1-0a6c-443f-9e67-358b71e01754" (UID: "a37f16b1-0a6c-443f-9e67-358b71e01754"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:32:02.903060 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.902839    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-home" (OuterVolumeSpecName: "home") pod "a37f16b1-0a6c-443f-9e67-358b71e01754" (UID: "a37f16b1-0a6c-443f-9e67-358b71e01754"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:32:02.903060 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.902935    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-home" (OuterVolumeSpecName: "home") pod "bdd4b573-6bc0-4a39-b09f-336c1a6cde78" (UID: "bdd4b573-6bc0-4a39-b09f-336c1a6cde78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:32:02.903524 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.903495    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdd4b573-6bc0-4a39-b09f-336c1a6cde78" (UID: "bdd4b573-6bc0-4a39-b09f-336c1a6cde78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:32:02.906012 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.905726    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdd4b573-6bc0-4a39-b09f-336c1a6cde78" (UID: "bdd4b573-6bc0-4a39-b09f-336c1a6cde78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:32:02.906630 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.906599    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a37f16b1-0a6c-443f-9e67-358b71e01754-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a37f16b1-0a6c-443f-9e67-358b71e01754" (UID: "a37f16b1-0a6c-443f-9e67-358b71e01754"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:32:02.919564 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.919286    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-dshm" (OuterVolumeSpecName: "dshm") pod "bdd4b573-6bc0-4a39-b09f-336c1a6cde78" (UID: "bdd4b573-6bc0-4a39-b09f-336c1a6cde78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:32:02.919564 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.919417    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-dshm" (OuterVolumeSpecName: "dshm") pod "a37f16b1-0a6c-443f-9e67-358b71e01754" (UID: "a37f16b1-0a6c-443f-9e67-358b71e01754"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:32:02.920120 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.920080    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37f16b1-0a6c-443f-9e67-358b71e01754-kube-api-access-xfzdj" (OuterVolumeSpecName: "kube-api-access-xfzdj") pod "a37f16b1-0a6c-443f-9e67-358b71e01754" (UID: "a37f16b1-0a6c-443f-9e67-358b71e01754"). InnerVolumeSpecName "kube-api-access-xfzdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:32:02.920120 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.920092    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-kube-api-access-npxm8" (OuterVolumeSpecName: "kube-api-access-npxm8") pod "bdd4b573-6bc0-4a39-b09f-336c1a6cde78" (UID: "bdd4b573-6bc0-4a39-b09f-336c1a6cde78"). InnerVolumeSpecName "kube-api-access-npxm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:32:02.923852 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.923804    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdd4b573-6bc0-4a39-b09f-336c1a6cde78" (UID: "bdd4b573-6bc0-4a39-b09f-336c1a6cde78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:32:02.930028 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.929959    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a37f16b1-0a6c-443f-9e67-358b71e01754" (UID: "a37f16b1-0a6c-443f-9e67-358b71e01754"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:32:02.967392 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.967351    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a37f16b1-0a6c-443f-9e67-358b71e01754" (UID: "a37f16b1-0a6c-443f-9e67-358b71e01754"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:32:02.998484 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:02.998413    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bdd4b573-6bc0-4a39-b09f-336c1a6cde78" (UID: "bdd4b573-6bc0-4a39-b09f-336c1a6cde78"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:32:03.003773 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.003732    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:32:03.003898 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.003780    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:32:03.003898 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.003796    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:32:03.003898 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.003812    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a37f16b1-0a6c-443f-9e67-358b71e01754-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:32:03.003898 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.003827    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfzdj\" (UniqueName: \"kubernetes.io/projected/a37f16b1-0a6c-443f-9e67-358b71e01754-kube-api-access-xfzdj\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:32:03.003898 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.003839    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:32:03.003898 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.003853    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:32:03.003898 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.003867    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:32:03.003898 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.003879    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:32:03.003898 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.003891    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npxm8\" (UniqueName: \"kubernetes.io/projected/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-kube-api-access-npxm8\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:32:03.004383 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.003922    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:32:03.004383 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.003938    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd4b573-6bc0-4a39-b09f-336c1a6cde78-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:32:03.004383 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.003950    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:32:03.004383 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.003977    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a37f16b1-0a6c-443f-9e67-358b71e01754-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:32:03.426763 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.426725    2570 generic.go:356] "Generic (PLEG): container finished" podID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerID="320b659e2905fa5592100d693ee2e702911ede9bb0bb2eef9c8f92ac80b72629" exitCode=137
Jul 29 17:32:03.427327 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.426805    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"
Jul 29 17:32:03.427327 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.426813    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" event={"ID":"bdd4b573-6bc0-4a39-b09f-336c1a6cde78","Type":"ContainerDied","Data":"320b659e2905fa5592100d693ee2e702911ede9bb0bb2eef9c8f92ac80b72629"}
Jul 29 17:32:03.427327 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.426893    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5" event={"ID":"bdd4b573-6bc0-4a39-b09f-336c1a6cde78","Type":"ContainerDied","Data":"31859851fd615a42cb41d6414a435debb8a41ba5ca08313fd4fe75012734e830"}
Jul 29 17:32:03.427327 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.426928    2570 scope.go:117] "RemoveContainer" containerID="320b659e2905fa5592100d693ee2e702911ede9bb0bb2eef9c8f92ac80b72629"
Jul 29 17:32:03.429787 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.429756    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn_a37f16b1-0a6c-443f-9e67-358b71e01754/main/0.log"
Jul 29 17:32:03.430968 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.430924    2570 generic.go:356] "Generic (PLEG): container finished" podID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerID="6feefb6bcaa30bf263e34abe1243f65fa3c539e3690bb741d94cc10acd4097a5" exitCode=137
Jul 29 17:32:03.431117 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.431033    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" event={"ID":"a37f16b1-0a6c-443f-9e67-358b71e01754","Type":"ContainerDied","Data":"6feefb6bcaa30bf263e34abe1243f65fa3c539e3690bb741d94cc10acd4097a5"}
Jul 29 17:32:03.431117 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.431103    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn" event={"ID":"a37f16b1-0a6c-443f-9e67-358b71e01754","Type":"ContainerDied","Data":"a9123025bb016ed9e48de29b6e3f6611772659c9a1b5c1bf784b5c79d94b56e3"}
Jul 29 17:32:03.431290 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.431247    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"
Jul 29 17:32:03.440806 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.440771    2570 scope.go:117] "RemoveContainer" containerID="41decbc305cce26a52612ea8345b360c1be29f202e85f9d273d69b19d731829e"
Jul 29 17:32:03.455893 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.455857    2570 scope.go:117] "RemoveContainer" containerID="320b659e2905fa5592100d693ee2e702911ede9bb0bb2eef9c8f92ac80b72629"
Jul 29 17:32:03.456736 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:32:03.456372    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"320b659e2905fa5592100d693ee2e702911ede9bb0bb2eef9c8f92ac80b72629\": container with ID starting with 320b659e2905fa5592100d693ee2e702911ede9bb0bb2eef9c8f92ac80b72629 not found: ID does not exist" containerID="320b659e2905fa5592100d693ee2e702911ede9bb0bb2eef9c8f92ac80b72629"
Jul 29 17:32:03.456736 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.456418    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"320b659e2905fa5592100d693ee2e702911ede9bb0bb2eef9c8f92ac80b72629"} err="failed to get container status \"320b659e2905fa5592100d693ee2e702911ede9bb0bb2eef9c8f92ac80b72629\": rpc error: code = NotFound desc = could not find container \"320b659e2905fa5592100d693ee2e702911ede9bb0bb2eef9c8f92ac80b72629\": container with ID starting with 320b659e2905fa5592100d693ee2e702911ede9bb0bb2eef9c8f92ac80b72629 not found: ID does not exist"
Jul 29 17:32:03.456736 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.456447    2570 scope.go:117] "RemoveContainer" containerID="41decbc305cce26a52612ea8345b360c1be29f202e85f9d273d69b19d731829e"
Jul 29 17:32:03.457604 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:32:03.457406    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41decbc305cce26a52612ea8345b360c1be29f202e85f9d273d69b19d731829e\": container with ID starting with 41decbc305cce26a52612ea8345b360c1be29f202e85f9d273d69b19d731829e not found: ID does not exist" containerID="41decbc305cce26a52612ea8345b360c1be29f202e85f9d273d69b19d731829e"
Jul 29 17:32:03.457604 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.457451    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41decbc305cce26a52612ea8345b360c1be29f202e85f9d273d69b19d731829e"} err="failed to get container status \"41decbc305cce26a52612ea8345b360c1be29f202e85f9d273d69b19d731829e\": rpc error: code = NotFound desc = could not find container \"41decbc305cce26a52612ea8345b360c1be29f202e85f9d273d69b19d731829e\": container with ID starting with 41decbc305cce26a52612ea8345b360c1be29f202e85f9d273d69b19d731829e not found: ID does not exist"
Jul 29 17:32:03.457604 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.457480    2570 scope.go:117] "RemoveContainer" containerID="6feefb6bcaa30bf263e34abe1243f65fa3c539e3690bb741d94cc10acd4097a5"
Jul 29 17:32:03.460123 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.460089    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"]
Jul 29 17:32:03.464433 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.464404    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bpczs5"]
Jul 29 17:32:03.469474 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.469443    2570 scope.go:117] "RemoveContainer" containerID="e285e91129eda121bf926012b378977a78e9e010b57f6c94f0e50b91151c3898"
Jul 29 17:32:03.475775 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.475721    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"]
Jul 29 17:32:03.482343 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.482282    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-9d6b6747-f45sn"]
Jul 29 17:32:03.559338 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.559315    2570 scope.go:117] "RemoveContainer" containerID="47c6f04458e4436b1c21bd1aa28f94c747f5c23eea843aa8903c2e47a7994952"
Jul 29 17:32:03.569939 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.569914    2570 scope.go:117] "RemoveContainer" containerID="6feefb6bcaa30bf263e34abe1243f65fa3c539e3690bb741d94cc10acd4097a5"
Jul 29 17:32:03.570347 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:32:03.570322    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6feefb6bcaa30bf263e34abe1243f65fa3c539e3690bb741d94cc10acd4097a5\": container with ID starting with 6feefb6bcaa30bf263e34abe1243f65fa3c539e3690bb741d94cc10acd4097a5 not found: ID does not exist" containerID="6feefb6bcaa30bf263e34abe1243f65fa3c539e3690bb741d94cc10acd4097a5"
Jul 29 17:32:03.570435 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.570359    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6feefb6bcaa30bf263e34abe1243f65fa3c539e3690bb741d94cc10acd4097a5"} err="failed to get container status \"6feefb6bcaa30bf263e34abe1243f65fa3c539e3690bb741d94cc10acd4097a5\": rpc error: code = NotFound desc = could not find container \"6feefb6bcaa30bf263e34abe1243f65fa3c539e3690bb741d94cc10acd4097a5\": container with ID starting with 6feefb6bcaa30bf263e34abe1243f65fa3c539e3690bb741d94cc10acd4097a5 not found: ID does not exist"
Jul 29 17:32:03.570435 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.570390    2570 scope.go:117] "RemoveContainer" containerID="e285e91129eda121bf926012b378977a78e9e010b57f6c94f0e50b91151c3898"
Jul 29 17:32:03.570716 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:32:03.570693    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e285e91129eda121bf926012b378977a78e9e010b57f6c94f0e50b91151c3898\": container with ID starting with e285e91129eda121bf926012b378977a78e9e010b57f6c94f0e50b91151c3898 not found: ID does not exist" containerID="e285e91129eda121bf926012b378977a78e9e010b57f6c94f0e50b91151c3898"
Jul 29 17:32:03.570816 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.570722    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e285e91129eda121bf926012b378977a78e9e010b57f6c94f0e50b91151c3898"} err="failed to get container status \"e285e91129eda121bf926012b378977a78e9e010b57f6c94f0e50b91151c3898\": rpc error: code = NotFound desc = could not find container \"e285e91129eda121bf926012b378977a78e9e010b57f6c94f0e50b91151c3898\": container with ID starting with e285e91129eda121bf926012b378977a78e9e010b57f6c94f0e50b91151c3898 not found: ID does not exist"
Jul 29 17:32:03.570816 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.570743    2570 scope.go:117] "RemoveContainer" containerID="47c6f04458e4436b1c21bd1aa28f94c747f5c23eea843aa8903c2e47a7994952"
Jul 29 17:32:03.571043 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:32:03.571019    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c6f04458e4436b1c21bd1aa28f94c747f5c23eea843aa8903c2e47a7994952\": container with ID starting with 47c6f04458e4436b1c21bd1aa28f94c747f5c23eea843aa8903c2e47a7994952 not found: ID does not exist" containerID="47c6f04458e4436b1c21bd1aa28f94c747f5c23eea843aa8903c2e47a7994952"
Jul 29 17:32:03.571091 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.571050    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c6f04458e4436b1c21bd1aa28f94c747f5c23eea843aa8903c2e47a7994952"} err="failed to get container status \"47c6f04458e4436b1c21bd1aa28f94c747f5c23eea843aa8903c2e47a7994952\": rpc error: code = NotFound desc = could not find container \"47c6f04458e4436b1c21bd1aa28f94c747f5c23eea843aa8903c2e47a7994952\": container with ID starting with 47c6f04458e4436b1c21bd1aa28f94c747f5c23eea843aa8903c2e47a7994952 not found: ID does not exist"
Jul 29 17:32:03.917399 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.917312    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" path="/var/lib/kubelet/pods/a37f16b1-0a6c-443f-9e67-358b71e01754/volumes"
Jul 29 17:32:03.918396 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:03.918372    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" path="/var/lib/kubelet/pods/bdd4b573-6bc0-4a39-b09f-336c1a6cde78/volumes"
Jul 29 17:32:05.065403 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:05.065360    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 17:32:09.831257 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:09.831209    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:32:09.846851 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:09.846806    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:32:15.065881 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:15.065822    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 17:32:19.830887 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:19.830836    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:32:19.846923 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:19.846876    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:32:25.066056 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:25.065994    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 17:32:29.830942 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:29.830897    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:32:29.846182 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:29.846118    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:32:35.065380 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:35.065330    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 17:32:39.831615 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:39.831555    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:32:39.846335 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:39.846286    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:32:45.065804 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:45.065745    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 17:32:49.831603 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:49.831537    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:32:49.846193 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:49.846138    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:32:55.065923 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:55.065880    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 17:32:59.831316 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:59.831266    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:32:59.846593 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:32:59.846541    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:33:05.065696 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:05.065584    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 17:33:09.831013 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:09.830968    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:33:09.846015 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:09.845977    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:33:15.076056 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:15.076023    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:33:15.084125 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:15.084098    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:33:19.831229 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:19.831137    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:33:19.846647 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:19.846594    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:33:29.831390 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:29.831334    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:33:29.846765 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:29.846713    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:33:32.550102 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:32.550052    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 29 17:33:32.550618 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:32.550450    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main" containerID="cri-o://62d2c12a2208a37fe04bd46837d8333ca3e708c5c06bfb96263437881c046cc7" gracePeriod=30
Jul 29 17:33:34.029944 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.029893    2570 generic.go:356] "Generic (PLEG): container finished" podID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerID="62d2c12a2208a37fe04bd46837d8333ca3e708c5c06bfb96263437881c046cc7" exitCode=0
Jul 29 17:33:34.029944 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.029936    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"ba0c65d5-ed23-442e-9779-076f89fd7280","Type":"ContainerDied","Data":"62d2c12a2208a37fe04bd46837d8333ca3e708c5c06bfb96263437881c046cc7"}
Jul 29 17:33:34.136466 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.136390    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:33:34.202234 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.202184    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-tmp-dir\") pod \"ba0c65d5-ed23-442e-9779-076f89fd7280\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") "
Jul 29 17:33:34.202446 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.202243    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-dshm\") pod \"ba0c65d5-ed23-442e-9779-076f89fd7280\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") "
Jul 29 17:33:34.202446 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.202269    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-home\") pod \"ba0c65d5-ed23-442e-9779-076f89fd7280\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") "
Jul 29 17:33:34.202446 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.202297    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba0c65d5-ed23-442e-9779-076f89fd7280-tls-certs\") pod \"ba0c65d5-ed23-442e-9779-076f89fd7280\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") "
Jul 29 17:33:34.202446 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.202329    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-kserve-provision-location\") pod \"ba0c65d5-ed23-442e-9779-076f89fd7280\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") "
Jul 29 17:33:34.202446 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.202409    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-model-cache\") pod \"ba0c65d5-ed23-442e-9779-076f89fd7280\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") "
Jul 29 17:33:34.202735 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.202459    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pdlk\" (UniqueName: \"kubernetes.io/projected/ba0c65d5-ed23-442e-9779-076f89fd7280-kube-api-access-9pdlk\") pod \"ba0c65d5-ed23-442e-9779-076f89fd7280\" (UID: \"ba0c65d5-ed23-442e-9779-076f89fd7280\") "
Jul 29 17:33:34.202836 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.202797    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba0c65d5-ed23-442e-9779-076f89fd7280" (UID: "ba0c65d5-ed23-442e-9779-076f89fd7280"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:33:34.203008 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.202981    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-home" (OuterVolumeSpecName: "home") pod "ba0c65d5-ed23-442e-9779-076f89fd7280" (UID: "ba0c65d5-ed23-442e-9779-076f89fd7280"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:33:34.205809 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.205759    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0c65d5-ed23-442e-9779-076f89fd7280-kube-api-access-9pdlk" (OuterVolumeSpecName: "kube-api-access-9pdlk") pod "ba0c65d5-ed23-442e-9779-076f89fd7280" (UID: "ba0c65d5-ed23-442e-9779-076f89fd7280"). InnerVolumeSpecName "kube-api-access-9pdlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:33:34.206100 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.206060    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-dshm" (OuterVolumeSpecName: "dshm") pod "ba0c65d5-ed23-442e-9779-076f89fd7280" (UID: "ba0c65d5-ed23-442e-9779-076f89fd7280"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:33:34.208105 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.208069    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba0c65d5-ed23-442e-9779-076f89fd7280-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba0c65d5-ed23-442e-9779-076f89fd7280" (UID: "ba0c65d5-ed23-442e-9779-076f89fd7280"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:33:34.223270 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.223223    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba0c65d5-ed23-442e-9779-076f89fd7280" (UID: "ba0c65d5-ed23-442e-9779-076f89fd7280"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:33:34.254600 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.254560    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ba0c65d5-ed23-442e-9779-076f89fd7280" (UID: "ba0c65d5-ed23-442e-9779-076f89fd7280"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:33:34.304149 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.304109    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:33:34.304149 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.304143    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:33:34.304149 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.304152    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:33:34.304149 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.304173    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba0c65d5-ed23-442e-9779-076f89fd7280-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:33:34.304443 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.304183    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:33:34.304443 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.304191    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba0c65d5-ed23-442e-9779-076f89fd7280-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:33:34.304443 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:34.304201    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pdlk\" (UniqueName: \"kubernetes.io/projected/ba0c65d5-ed23-442e-9779-076f89fd7280-kube-api-access-9pdlk\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:33:35.038482 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:35.038451    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 29 17:33:35.039088 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:35.038481    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"ba0c65d5-ed23-442e-9779-076f89fd7280","Type":"ContainerDied","Data":"6eeb5c4dc4534480fb388618184a41e9a8837b28bde597758ca137a03dfabbd1"}
Jul 29 17:33:35.039088 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:35.038551    2570 scope.go:117] "RemoveContainer" containerID="62d2c12a2208a37fe04bd46837d8333ca3e708c5c06bfb96263437881c046cc7"
Jul 29 17:33:35.049599 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:35.049582    2570 scope.go:117] "RemoveContainer" containerID="9bcbc19ce3e21c1799f84600d28771270ca9ba85bf3589ffadf22af113b6284a"
Jul 29 17:33:35.063023 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:35.062997    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 29 17:33:35.068867 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:35.068839    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 29 17:33:35.914239 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:35.914204    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" path="/var/lib/kubelet/pods/ba0c65d5-ed23-442e-9779-076f89fd7280/volumes"
Jul 29 17:33:39.830956 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:39.830905    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:33:39.846216 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:39.846152    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:33:49.830843 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:49.830775    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:33:49.846439 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:49.846395    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:33:59.830772 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:59.830719    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:33:59.846546 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:33:59.846512    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:34:09.831376 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:09.831321    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:34:09.846057 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:09.846015    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:34:19.831604 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:19.831549    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Jul 29 17:34:19.846516 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:19.846474    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 17:34:29.840482 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:29.840449    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:34:29.853669 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:29.853637    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:34:29.855958 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:29.855936    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:34:29.864046 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:29.864019    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:34:42.917831 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:42.917795    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"]
Jul 29 17:34:42.918486 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:42.918455    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main" containerID="cri-o://0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c" gracePeriod=30
Jul 29 17:34:42.922131 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:42.922096    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"]
Jul 29 17:34:42.922405 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:42.922379    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main" containerID="cri-o://3b121309b807cea8f9d52067430791648b76e5fcc4d36fb9d630a5cc51f7f70e" gracePeriod=30
Jul 29 17:34:59.701291 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:59.701231    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:34:59.718377 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:59.718344    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/llm-d-routing-sidecar/0.log"
Jul 29 17:34:59.728613 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:59.728588    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/storage-initializer/0.log"
Jul 29 17:34:59.746891 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:59.746858    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/main/0.log"
Jul 29 17:34:59.753779 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:59.753753    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/storage-initializer/0.log"
Jul 29 17:34:59.785859 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:34:59.785834    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7b76d7c4bf-4txdd_35c2292a-edbb-4c85-8044-192149996162/istio-proxy/0.log"
Jul 29 17:35:00.727195 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:00.727143    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:00.734990 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:00.734961    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/llm-d-routing-sidecar/0.log"
Jul 29 17:35:00.744861 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:00.744828    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/storage-initializer/0.log"
Jul 29 17:35:00.764362 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:00.764332    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/main/0.log"
Jul 29 17:35:00.771430 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:00.771403    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/storage-initializer/0.log"
Jul 29 17:35:00.803938 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:00.803902    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7b76d7c4bf-4txdd_35c2292a-edbb-4c85-8044-192149996162/istio-proxy/0.log"
Jul 29 17:35:01.769632 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:01.769597    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:01.777269 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:01.777236    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/llm-d-routing-sidecar/0.log"
Jul 29 17:35:01.787005 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:01.786973    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/storage-initializer/0.log"
Jul 29 17:35:01.806788 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:01.806748    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/main/0.log"
Jul 29 17:35:01.813476 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:01.813450    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/storage-initializer/0.log"
Jul 29 17:35:01.845717 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:01.845661    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7b76d7c4bf-4txdd_35c2292a-edbb-4c85-8044-192149996162/istio-proxy/0.log"
Jul 29 17:35:02.795982 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:02.795942    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:02.804322 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:02.804296    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/llm-d-routing-sidecar/0.log"
Jul 29 17:35:02.816559 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:02.816518    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/storage-initializer/0.log"
Jul 29 17:35:02.835812 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:02.835775    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/main/0.log"
Jul 29 17:35:02.843050 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:02.843016    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/storage-initializer/0.log"
Jul 29 17:35:02.875799 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:02.875765    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7b76d7c4bf-4txdd_35c2292a-edbb-4c85-8044-192149996162/istio-proxy/0.log"
Jul 29 17:35:03.823727 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:03.823697    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:03.830847 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:03.830819    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/llm-d-routing-sidecar/0.log"
Jul 29 17:35:03.841017 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:03.840986    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/storage-initializer/0.log"
Jul 29 17:35:03.860352 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:03.860325    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/main/0.log"
Jul 29 17:35:03.866872 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:03.866848    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/storage-initializer/0.log"
Jul 29 17:35:03.897641 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:03.897603    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7b76d7c4bf-4txdd_35c2292a-edbb-4c85-8044-192149996162/istio-proxy/0.log"
Jul 29 17:35:04.842946 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:04.842900    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:04.851465 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:04.851441    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/llm-d-routing-sidecar/0.log"
Jul 29 17:35:04.862654 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:04.862621    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/storage-initializer/0.log"
Jul 29 17:35:04.881929 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:04.881897    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/main/0.log"
Jul 29 17:35:04.889792 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:04.889769    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/storage-initializer/0.log"
Jul 29 17:35:04.933404 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:04.933376    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7b76d7c4bf-4txdd_35c2292a-edbb-4c85-8044-192149996162/istio-proxy/0.log"
Jul 29 17:35:05.885219 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:05.885186    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:05.893849 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:05.893821    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/llm-d-routing-sidecar/0.log"
Jul 29 17:35:05.904910 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:05.904884    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/storage-initializer/0.log"
Jul 29 17:35:05.930234 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:05.930197    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/main/0.log"
Jul 29 17:35:05.938269 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:05.938240    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/storage-initializer/0.log"
Jul 29 17:35:05.974092 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:05.974042    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7b76d7c4bf-4txdd_35c2292a-edbb-4c85-8044-192149996162/istio-proxy/0.log"
Jul 29 17:35:06.920134 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:06.920105    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:06.927908 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:06.927882    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/llm-d-routing-sidecar/0.log"
Jul 29 17:35:06.939215 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:06.939195    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/storage-initializer/0.log"
Jul 29 17:35:06.959137 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:06.959109    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/main/0.log"
Jul 29 17:35:06.965423 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:06.965397    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/storage-initializer/0.log"
Jul 29 17:35:06.995487 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:06.995452    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7b76d7c4bf-4txdd_35c2292a-edbb-4c85-8044-192149996162/istio-proxy/0.log"
Jul 29 17:35:07.930299 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:07.930264    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:07.940509 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:07.940480    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/llm-d-routing-sidecar/0.log"
Jul 29 17:35:07.951279 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:07.951244    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/storage-initializer/0.log"
Jul 29 17:35:07.972907 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:07.972877    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/main/0.log"
Jul 29 17:35:07.980718 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:07.980683    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/storage-initializer/0.log"
Jul 29 17:35:08.017508 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:08.017459    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7b76d7c4bf-4txdd_35c2292a-edbb-4c85-8044-192149996162/istio-proxy/0.log"
Jul 29 17:35:08.960301 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:08.960264    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:08.968563 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:08.968539    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/llm-d-routing-sidecar/0.log"
Jul 29 17:35:08.978958 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:08.978933    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/storage-initializer/0.log"
Jul 29 17:35:08.998277 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:08.998251    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/main/0.log"
Jul 29 17:35:09.006042 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:09.006014    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/storage-initializer/0.log"
Jul 29 17:35:09.039143 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:09.039106    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7b76d7c4bf-4txdd_35c2292a-edbb-4c85-8044-192149996162/istio-proxy/0.log"
Jul 29 17:35:10.005769 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:10.005734    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:10.014125 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:10.014101    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/llm-d-routing-sidecar/0.log"
Jul 29 17:35:10.024140 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:10.024106    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/storage-initializer/0.log"
Jul 29 17:35:10.044707 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:10.044676    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/main/0.log"
Jul 29 17:35:10.054694 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:10.054666    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/storage-initializer/0.log"
Jul 29 17:35:10.091624 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:10.091597    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7b76d7c4bf-4txdd_35c2292a-edbb-4c85-8044-192149996162/istio-proxy/0.log"
Jul 29 17:35:11.085350 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:11.085317    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:11.093344 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:11.093310    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/llm-d-routing-sidecar/0.log"
Jul 29 17:35:11.103795 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:11.103771    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/storage-initializer/0.log"
Jul 29 17:35:11.123952 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:11.123926    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/main/0.log"
Jul 29 17:35:11.131620 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:11.131595    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/storage-initializer/0.log"
Jul 29 17:35:11.164680 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:11.164652    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7b76d7c4bf-4txdd_35c2292a-edbb-4c85-8044-192149996162/istio-proxy/0.log"
Jul 29 17:35:12.221112 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:12.221081    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:12.230560 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:12.230533    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/llm-d-routing-sidecar/0.log"
Jul 29 17:35:12.243845 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:12.243816    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/storage-initializer/0.log"
Jul 29 17:35:12.265706 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:12.265668    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/main/0.log"
Jul 29 17:35:12.275141 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:12.275105    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/storage-initializer/0.log"
Jul 29 17:35:12.308848 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:12.308817    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7b76d7c4bf-4txdd_35c2292a-edbb-4c85-8044-192149996162/istio-proxy/0.log"
Jul 29 17:35:12.919121 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:12.919075    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="llm-d-routing-sidecar" containerID="cri-o://792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197" gracePeriod=2
Jul 29 17:35:13.294648 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.294614    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:13.302637 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.302609    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/llm-d-routing-sidecar/0.log"
Jul 29 17:35:13.313419 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.313383    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/storage-initializer/0.log"
Jul 29 17:35:13.340771 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.340693    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/main/0.log"
Jul 29 17:35:13.347742 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.347715    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp_33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/storage-initializer/0.log"
Jul 29 17:35:13.384524 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.384489    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7b76d7c4bf-4txdd_35c2292a-edbb-4c85-8044-192149996162/istio-proxy/0.log"
Jul 29 17:35:13.390260 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.390240    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:13.393419 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.393397    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:35:13.395805 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.395788    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:35:13.498735 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.498638    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-kserve-provision-location\") pod \"1f55dce6-b3a0-4631-ba6a-a77936e41175\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") "
Jul 29 17:35:13.498735 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.498678    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7sfz\" (UniqueName: \"kubernetes.io/projected/1f55dce6-b3a0-4631-ba6a-a77936e41175-kube-api-access-k7sfz\") pod \"1f55dce6-b3a0-4631-ba6a-a77936e41175\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") "
Jul 29 17:35:13.498735 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.498699    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-dshm\") pod \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") "
Jul 29 17:35:13.498735 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.498717    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-model-cache\") pod \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") "
Jul 29 17:35:13.499090 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.498743    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-tmp-dir\") pod \"1f55dce6-b3a0-4631-ba6a-a77936e41175\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") "
Jul 29 17:35:13.499090 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.498771    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f55dce6-b3a0-4631-ba6a-a77936e41175-tls-certs\") pod \"1f55dce6-b3a0-4631-ba6a-a77936e41175\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") "
Jul 29 17:35:13.499090 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.498815    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k92z\" (UniqueName: \"kubernetes.io/projected/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-kube-api-access-2k92z\") pod \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") "
Jul 29 17:35:13.499090 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.498842    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-model-cache\") pod \"1f55dce6-b3a0-4631-ba6a-a77936e41175\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") "
Jul 29 17:35:13.499090 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.498865    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-dshm\") pod \"1f55dce6-b3a0-4631-ba6a-a77936e41175\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") "
Jul 29 17:35:13.499090 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.499008    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-model-cache" (OuterVolumeSpecName: "model-cache") pod "33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" (UID: "33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:35:13.499465 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.499403    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f55dce6-b3a0-4631-ba6a-a77936e41175" (UID: "1f55dce6-b3a0-4631-ba6a-a77936e41175"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:35:13.499550 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.499529    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-tls-certs\") pod \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") "
Jul 29 17:35:13.499610 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.499597    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-home\") pod \"1f55dce6-b3a0-4631-ba6a-a77936e41175\" (UID: \"1f55dce6-b3a0-4631-ba6a-a77936e41175\") "
Jul 29 17:35:13.500905 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.499671    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-home\") pod \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") "
Jul 29 17:35:13.500905 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.499708    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-tmp-dir\") pod \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") "
Jul 29 17:35:13.500905 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.499737    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-kserve-provision-location\") pod \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\" (UID: \"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c\") "
Jul 29 17:35:13.500905 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.500079    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:35:13.500905 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.500097    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:35:13.500905 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.500477    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-home" (OuterVolumeSpecName: "home") pod "1f55dce6-b3a0-4631-ba6a-a77936e41175" (UID: "1f55dce6-b3a0-4631-ba6a-a77936e41175"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:35:13.500905 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.500892    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-home" (OuterVolumeSpecName: "home") pod "33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" (UID: "33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:35:13.501572 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.501545    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-dshm" (OuterVolumeSpecName: "dshm") pod "1f55dce6-b3a0-4631-ba6a-a77936e41175" (UID: "1f55dce6-b3a0-4631-ba6a-a77936e41175"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:35:13.501661 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.501553    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-dshm" (OuterVolumeSpecName: "dshm") pod "33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" (UID: "33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:35:13.501661 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.501571    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-kube-api-access-2k92z" (OuterVolumeSpecName: "kube-api-access-2k92z") pod "33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" (UID: "33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c"). InnerVolumeSpecName "kube-api-access-2k92z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:35:13.502379 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.502353    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f55dce6-b3a0-4631-ba6a-a77936e41175-kube-api-access-k7sfz" (OuterVolumeSpecName: "kube-api-access-k7sfz") pod "1f55dce6-b3a0-4631-ba6a-a77936e41175" (UID: "1f55dce6-b3a0-4631-ba6a-a77936e41175"). InnerVolumeSpecName "kube-api-access-k7sfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:35:13.502986 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.502963    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f55dce6-b3a0-4631-ba6a-a77936e41175-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f55dce6-b3a0-4631-ba6a-a77936e41175" (UID: "1f55dce6-b3a0-4631-ba6a-a77936e41175"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:35:13.503212 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.503194    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" (UID: "33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:35:13.519874 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.519832    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f55dce6-b3a0-4631-ba6a-a77936e41175" (UID: "1f55dce6-b3a0-4631-ba6a-a77936e41175"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:35:13.527734 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.527675    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" (UID: "33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:35:13.551788 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.551749    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1f55dce6-b3a0-4631-ba6a-a77936e41175" (UID: "1f55dce6-b3a0-4631-ba6a-a77936e41175"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:35:13.559619 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.559573    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" (UID: "33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:35:13.600811 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.600769    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:35:13.600971 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.600828    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:35:13.600971 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.600845    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:35:13.600971 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.600856    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:35:13.600971 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.600870    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7sfz\" (UniqueName: \"kubernetes.io/projected/1f55dce6-b3a0-4631-ba6a-a77936e41175-kube-api-access-k7sfz\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:35:13.600971 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.600882    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:35:13.600971 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.600893    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:35:13.600971 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.600905    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f55dce6-b3a0-4631-ba6a-a77936e41175-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:35:13.600971 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.600916    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k92z\" (UniqueName: \"kubernetes.io/projected/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-kube-api-access-2k92z\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:35:13.600971 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.600929    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:35:13.600971 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.600941    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:35:13.600971 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.600951    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f55dce6-b3a0-4631-ba6a-a77936e41175-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Jul 29 17:35:13.620575 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.620542    2570 generic.go:356] "Generic (PLEG): container finished" podID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerID="3b121309b807cea8f9d52067430791648b76e5fcc4d36fb9d630a5cc51f7f70e" exitCode=137
Jul 29 17:35:13.620760 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.620593    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"
Jul 29 17:35:13.620760 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.620622    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" event={"ID":"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c","Type":"ContainerDied","Data":"3b121309b807cea8f9d52067430791648b76e5fcc4d36fb9d630a5cc51f7f70e"}
Jul 29 17:35:13.620760 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.620673    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp" event={"ID":"33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c","Type":"ContainerDied","Data":"3b920dd80da2b469ba2f98a283efe4d4024da57aef23044862ad5f6c1f0d3b0a"}
Jul 29 17:35:13.620760 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.620699    2570 scope.go:117] "RemoveContainer" containerID="3b121309b807cea8f9d52067430791648b76e5fcc4d36fb9d630a5cc51f7f70e"
Jul 29 17:35:13.622189 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.622157    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw_1f55dce6-b3a0-4631-ba6a-a77936e41175/main/0.log"
Jul 29 17:35:13.622936 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.622901    2570 generic.go:356] "Generic (PLEG): container finished" podID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerID="0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c" exitCode=137
Jul 29 17:35:13.622936 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.622925    2570 generic.go:356] "Generic (PLEG): container finished" podID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerID="792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197" exitCode=0
Jul 29 17:35:13.623075 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.622938    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" event={"ID":"1f55dce6-b3a0-4631-ba6a-a77936e41175","Type":"ContainerDied","Data":"0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c"}
Jul 29 17:35:13.623075 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.622977    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"
Jul 29 17:35:13.623075 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.622997    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" event={"ID":"1f55dce6-b3a0-4631-ba6a-a77936e41175","Type":"ContainerDied","Data":"792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197"}
Jul 29 17:35:13.623075 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.623028    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw" event={"ID":"1f55dce6-b3a0-4631-ba6a-a77936e41175","Type":"ContainerDied","Data":"943663e1155980fcca090f11efa3c2de1b229e3948a4af2cd6e8be0d0a453dee"}
Jul 29 17:35:13.630530 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.630511    2570 scope.go:117] "RemoveContainer" containerID="3c16782601b0ce9e3188e236854b5865e76115d04dab6811db3502b6591eecdf"
Jul 29 17:35:13.642123 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.642104    2570 scope.go:117] "RemoveContainer" containerID="3b121309b807cea8f9d52067430791648b76e5fcc4d36fb9d630a5cc51f7f70e"
Jul 29 17:35:13.642419 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:35:13.642399    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b121309b807cea8f9d52067430791648b76e5fcc4d36fb9d630a5cc51f7f70e\": container with ID starting with 3b121309b807cea8f9d52067430791648b76e5fcc4d36fb9d630a5cc51f7f70e not found: ID does not exist" containerID="3b121309b807cea8f9d52067430791648b76e5fcc4d36fb9d630a5cc51f7f70e"
Jul 29 17:35:13.642468 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.642429    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b121309b807cea8f9d52067430791648b76e5fcc4d36fb9d630a5cc51f7f70e"} err="failed to get container status \"3b121309b807cea8f9d52067430791648b76e5fcc4d36fb9d630a5cc51f7f70e\": rpc error: code = NotFound desc = could not find container \"3b121309b807cea8f9d52067430791648b76e5fcc4d36fb9d630a5cc51f7f70e\": container with ID starting with 3b121309b807cea8f9d52067430791648b76e5fcc4d36fb9d630a5cc51f7f70e not found: ID does not exist"
Jul 29 17:35:13.642468 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.642447    2570 scope.go:117] "RemoveContainer" containerID="3c16782601b0ce9e3188e236854b5865e76115d04dab6811db3502b6591eecdf"
Jul 29 17:35:13.642673 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:35:13.642655    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c16782601b0ce9e3188e236854b5865e76115d04dab6811db3502b6591eecdf\": container with ID starting with 3c16782601b0ce9e3188e236854b5865e76115d04dab6811db3502b6591eecdf not found: ID does not exist" containerID="3c16782601b0ce9e3188e236854b5865e76115d04dab6811db3502b6591eecdf"
Jul 29 17:35:13.642715 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.642678    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c16782601b0ce9e3188e236854b5865e76115d04dab6811db3502b6591eecdf"} err="failed to get container status \"3c16782601b0ce9e3188e236854b5865e76115d04dab6811db3502b6591eecdf\": rpc error: code = NotFound desc = could not find container \"3c16782601b0ce9e3188e236854b5865e76115d04dab6811db3502b6591eecdf\": container with ID starting with 3c16782601b0ce9e3188e236854b5865e76115d04dab6811db3502b6591eecdf not found: ID does not exist"
Jul 29 17:35:13.642715 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.642693    2570 scope.go:117] "RemoveContainer" containerID="0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c"
Jul 29 17:35:13.646545 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.646524    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"]
Jul 29 17:35:13.651688 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.651662    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-965555885-9x9pp"]
Jul 29 17:35:13.653506 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.653489    2570 scope.go:117] "RemoveContainer" containerID="314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398"
Jul 29 17:35:13.663755 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.663726    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"]
Jul 29 17:35:13.665213 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.665193    2570 scope.go:117] "RemoveContainer" containerID="792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197"
Jul 29 17:35:13.668023 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.668000    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-66f45fb4cd-vl9fw"]
Jul 29 17:35:13.675477 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.675450    2570 scope.go:117] "RemoveContainer" containerID="0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c"
Jul 29 17:35:13.675776 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:35:13.675756    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c\": container with ID starting with 0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c not found: ID does not exist" containerID="0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c"
Jul 29 17:35:13.675834 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.675787    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c"} err="failed to get container status \"0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c\": rpc error: code = NotFound desc = could not find container \"0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c\": container with ID starting with 0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c not found: ID does not exist"
Jul 29 17:35:13.675834 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.675808    2570 scope.go:117] "RemoveContainer" containerID="314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398"
Jul 29 17:35:13.676054 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:35:13.676041    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398\": container with ID starting with 314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398 not found: ID does not exist" containerID="314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398"
Jul 29 17:35:13.676099 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.676057    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398"} err="failed to get container status \"314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398\": rpc error: code = NotFound desc = could not find container \"314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398\": container with ID starting with 314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398 not found: ID does not exist"
Jul 29 17:35:13.676099 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.676070    2570 scope.go:117] "RemoveContainer" containerID="792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197"
Jul 29 17:35:13.676332 ip-10-0-134-156 kubenswrapper[2570]: E0729 17:35:13.676318    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197\": container with ID starting with 792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197 not found: ID does not exist" containerID="792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197"
Jul 29 17:35:13.676383 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.676335    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197"} err="failed to get container status \"792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197\": rpc error: code = NotFound desc = could not find container \"792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197\": container with ID starting with 792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197 not found: ID does not exist"
Jul 29 17:35:13.676383 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.676348    2570 scope.go:117] "RemoveContainer" containerID="0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c"
Jul 29 17:35:13.676559 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.676539    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c"} err="failed to get container status \"0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c\": rpc error: code = NotFound desc = could not find container \"0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c\": container with ID starting with 0c9bd560b7d8ac226b017a27042711219ffc1bdcac5d83ead363864b0319965c not found: ID does not exist"
Jul 29 17:35:13.676602 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.676560    2570 scope.go:117] "RemoveContainer" containerID="314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398"
Jul 29 17:35:13.676767 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.676752    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398"} err="failed to get container status \"314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398\": rpc error: code = NotFound desc = could not find container \"314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398\": container with ID starting with 314cb5c9645418debd57c5ae2a2518820e7d246b5a6c2a19513f25ec1a3d0398 not found: ID does not exist"
Jul 29 17:35:13.676811 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.676769    2570 scope.go:117] "RemoveContainer" containerID="792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197"
Jul 29 17:35:13.677020 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.676998    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197"} err="failed to get container status \"792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197\": rpc error: code = NotFound desc = could not find container \"792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197\": container with ID starting with 792826e8924b6bebc20269493f2ca9dca5f6d0eab61127c3cb4bdfc0db852197 not found: ID does not exist"
Jul 29 17:35:13.914746 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.914714    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" path="/var/lib/kubelet/pods/1f55dce6-b3a0-4631-ba6a-a77936e41175/volumes"
Jul 29 17:35:13.915200 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:13.915186    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" path="/var/lib/kubelet/pods/33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c/volumes"
Jul 29 17:35:14.401958 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:14.401923    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5df7c8cd4c-mgz6p_fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6/router/0.log"
Jul 29 17:35:15.229323 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:15.229292    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5df7c8cd4c-mgz6p_fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6/router/0.log"
Jul 29 17:35:16.010870 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:16.010838    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-94qdj_f62182af-14ff-4795-b4fd-34cf1ae2e435/manager/0.log"
Jul 29 17:35:16.089589 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:16.089552    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-q7znc_8d565869-182a-4c65-97a7-94053de639ef/limitador/0.log"
Jul 29 17:35:16.104410 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:16.104378    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-pc9v2_0cc85d6c-1484-4066-b303-a2d2ae732521/manager/0.log"
Jul 29 17:35:18.430398 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430360    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4lvf9/must-gather-lkws2"]
Jul 29 17:35:18.430825 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430808    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="storage-initializer"
Jul 29 17:35:18.430868 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430828    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="storage-initializer"
Jul 29 17:35:18.430868 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430840    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="storage-initializer"
Jul 29 17:35:18.430868 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430845    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="storage-initializer"
Jul 29 17:35:18.430868 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430853    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main"
Jul 29 17:35:18.430868 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430860    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main"
Jul 29 17:35:18.430868 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430868    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="storage-initializer"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430874    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="storage-initializer"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430887    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="llm-d-routing-sidecar"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430893    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="llm-d-routing-sidecar"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430899    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="llm-d-routing-sidecar"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430905    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="llm-d-routing-sidecar"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430911    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430916    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430922    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="storage-initializer"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430926    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="storage-initializer"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430932    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430937    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430962    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="storage-initializer"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430968    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="storage-initializer"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430974    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430979    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430989    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.430994    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.431049    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="llm-d-routing-sidecar"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.431057    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd4b573-6bc0-4a39-b09f-336c1a6cde78" containerName="main"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.431063    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="llm-d-routing-sidecar"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.431069    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f55dce6-b3a0-4631-ba6a-a77936e41175" containerName="main"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.431074    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba0c65d5-ed23-442e-9779-076f89fd7280" containerName="main"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.431080    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a37f16b1-0a6c-443f-9e67-358b71e01754" containerName="main"
Jul 29 17:35:18.431079 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.431086    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ee2bb9-7aca-4e66-96e0-d8bcaa6f557c" containerName="main"
Jul 29 17:35:18.436729 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.436705    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4lvf9/must-gather-lkws2"
Jul 29 17:35:18.439904 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.439871    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4lvf9\"/\"default-dockercfg-bg4rw\""
Jul 29 17:35:18.439904 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.439871    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4lvf9\"/\"kube-root-ca.crt\""
Jul 29 17:35:18.440658 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.440639    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4lvf9\"/\"openshift-service-ca.crt\""
Jul 29 17:35:18.447111 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.447080    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4lvf9/must-gather-lkws2"]
Jul 29 17:35:18.546082 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.546043    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2ef851b4-a9df-41a2-b5ee-e120cc428dce-must-gather-output\") pod \"must-gather-lkws2\" (UID: \"2ef851b4-a9df-41a2-b5ee-e120cc428dce\") " pod="openshift-must-gather-4lvf9/must-gather-lkws2"
Jul 29 17:35:18.546299 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.546211    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9h2\" (UniqueName: \"kubernetes.io/projected/2ef851b4-a9df-41a2-b5ee-e120cc428dce-kube-api-access-fg9h2\") pod \"must-gather-lkws2\" (UID: \"2ef851b4-a9df-41a2-b5ee-e120cc428dce\") " pod="openshift-must-gather-4lvf9/must-gather-lkws2"
Jul 29 17:35:18.647409 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.647356    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9h2\" (UniqueName: \"kubernetes.io/projected/2ef851b4-a9df-41a2-b5ee-e120cc428dce-kube-api-access-fg9h2\") pod \"must-gather-lkws2\" (UID: \"2ef851b4-a9df-41a2-b5ee-e120cc428dce\") " pod="openshift-must-gather-4lvf9/must-gather-lkws2"
Jul 29 17:35:18.647596 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.647436    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2ef851b4-a9df-41a2-b5ee-e120cc428dce-must-gather-output\") pod \"must-gather-lkws2\" (UID: \"2ef851b4-a9df-41a2-b5ee-e120cc428dce\") " pod="openshift-must-gather-4lvf9/must-gather-lkws2"
Jul 29 17:35:18.647898 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.647878    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2ef851b4-a9df-41a2-b5ee-e120cc428dce-must-gather-output\") pod \"must-gather-lkws2\" (UID: \"2ef851b4-a9df-41a2-b5ee-e120cc428dce\") " pod="openshift-must-gather-4lvf9/must-gather-lkws2"
Jul 29 17:35:18.655937 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.655909    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9h2\" (UniqueName: \"kubernetes.io/projected/2ef851b4-a9df-41a2-b5ee-e120cc428dce-kube-api-access-fg9h2\") pod \"must-gather-lkws2\" (UID: \"2ef851b4-a9df-41a2-b5ee-e120cc428dce\") " pod="openshift-must-gather-4lvf9/must-gather-lkws2"
Jul 29 17:35:18.747085 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.746996    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4lvf9/must-gather-lkws2"
Jul 29 17:35:18.889419 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.889387    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4lvf9/must-gather-lkws2"]
Jul 29 17:35:18.891018 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:35:18.890995    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ef851b4_a9df_41a2_b5ee_e120cc428dce.slice/crio-e15384aae28dfaabce6e4af98aabb6fa64b3c92031efb2d6dc8b690adb084c76 WatchSource:0}: Error finding container e15384aae28dfaabce6e4af98aabb6fa64b3c92031efb2d6dc8b690adb084c76: Status 404 returned error can't find the container with id e15384aae28dfaabce6e4af98aabb6fa64b3c92031efb2d6dc8b690adb084c76
Jul 29 17:35:18.892965 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:18.892946    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 17:35:19.659419 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:19.659353    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4lvf9/must-gather-lkws2" event={"ID":"2ef851b4-a9df-41a2-b5ee-e120cc428dce","Type":"ContainerStarted","Data":"e15384aae28dfaabce6e4af98aabb6fa64b3c92031efb2d6dc8b690adb084c76"}
Jul 29 17:35:20.670823 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:20.670780    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4lvf9/must-gather-lkws2" event={"ID":"2ef851b4-a9df-41a2-b5ee-e120cc428dce","Type":"ContainerStarted","Data":"50b3e8314201fd8c1ce1b914227d5411c993ec59f90d6904d0fd37fb54d31ed8"}
Jul 29 17:35:20.671327 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:20.670837    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4lvf9/must-gather-lkws2" event={"ID":"2ef851b4-a9df-41a2-b5ee-e120cc428dce","Type":"ContainerStarted","Data":"367d97839e3da52d2b34f8711cdabdadf1423728f9a6e0b9453bca607d116b25"}
Jul 29 17:35:20.689113 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:20.689055    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4lvf9/must-gather-lkws2" podStartSLOduration=1.959040962 podStartE2EDuration="2.689033772s" totalImagesPullingTime="729.99281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:35:20.687329648 +0000 UTC m=+2649.176849317" watchObservedRunningTime="2026-07-29 17:35:20.689033772 +0000 UTC m=+2649.178553429"
Jul 29 17:35:21.292933 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:21.292902    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dxzh5_b1bd992f-213e-46bd-87a6-b06f7895719d/global-pull-secret-syncer/0.log"
Jul 29 17:35:21.470605 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:21.470570    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ttg74_f28253eb-90bd-48ad-ade7-a48e9fab2605/konnectivity-agent/0.log"
Jul 29 17:35:21.493136 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:21.493109    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-156.ec2.internal_159881cacdca307468786a6d29c1c03f/haproxy/0.log"
Jul 29 17:35:26.162034 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:26.161998    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-94qdj_f62182af-14ff-4795-b4fd-34cf1ae2e435/manager/0.log"
Jul 29 17:35:26.311237 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:26.311090    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-q7znc_8d565869-182a-4c65-97a7-94053de639ef/limitador/0.log"
Jul 29 17:35:26.364397 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:26.364357    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-pc9v2_0cc85d6c-1484-4066-b303-a2d2ae732521/manager/0.log"
Jul 29 17:35:27.994187 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:27.993490    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-787s9_8d895e18-c64e-4656-868b-4d5191988051/cluster-monitoring-operator/0.log"
Jul 29 17:35:28.348139 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:28.348107    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hmkc5_78ba15ea-514c-4528-b2e2-392ea1185d0d/node-exporter/0.log"
Jul 29 17:35:28.371306 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:28.371280    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hmkc5_78ba15ea-514c-4528-b2e2-392ea1185d0d/kube-rbac-proxy/0.log"
Jul 29 17:35:28.397011 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:28.396984    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hmkc5_78ba15ea-514c-4528-b2e2-392ea1185d0d/init-textfile/0.log"
Jul 29 17:35:30.464354 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:30.464316    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-d59pv_9a232c9c-55d3-4118-90cc-49aae6db92d5/networking-console-plugin/0.log"
Jul 29 17:35:31.005793 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:31.005756    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/1.log"
Jul 29 17:35:31.015655 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:31.015523    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-gg4gt_357c4ea3-b7a7-4f23-8ac7-f06745febc90/console-operator/2.log"
Jul 29 17:35:31.527366 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:31.527329    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-45wpw_162c70b6-13be-43f4-9b50-927f0a934e62/download-server/0.log"
Jul 29 17:35:32.031756 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.031708    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"]
Jul 29 17:35:32.039360 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.039325    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.044985 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.044954    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"]
Jul 29 17:35:32.151039 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.151012    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-5mnw6_50f45c1b-30e6-4965-8174-24bd4e0f815b/volume-data-source-validator/0.log"
Jul 29 17:35:32.183016 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.182968    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2444311a-c7ed-42b1-8388-3aa860716c5c-sys\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.183236 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.183056    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2444311a-c7ed-42b1-8388-3aa860716c5c-podres\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.183236 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.183087    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk8ps\" (UniqueName: \"kubernetes.io/projected/2444311a-c7ed-42b1-8388-3aa860716c5c-kube-api-access-xk8ps\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.183236 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.183133    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2444311a-c7ed-42b1-8388-3aa860716c5c-lib-modules\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.183236 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.183192    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2444311a-c7ed-42b1-8388-3aa860716c5c-proc\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.283984 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.283884    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2444311a-c7ed-42b1-8388-3aa860716c5c-proc\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.283984 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.283947    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2444311a-c7ed-42b1-8388-3aa860716c5c-sys\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.284217 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.283988    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2444311a-c7ed-42b1-8388-3aa860716c5c-podres\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.284217 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.284008    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk8ps\" (UniqueName: \"kubernetes.io/projected/2444311a-c7ed-42b1-8388-3aa860716c5c-kube-api-access-xk8ps\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.284217 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.284034    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2444311a-c7ed-42b1-8388-3aa860716c5c-lib-modules\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.284217 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.284040    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2444311a-c7ed-42b1-8388-3aa860716c5c-proc\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.284217 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.284093    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2444311a-c7ed-42b1-8388-3aa860716c5c-sys\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.284217 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.284157    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2444311a-c7ed-42b1-8388-3aa860716c5c-lib-modules\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.284217 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.284203    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2444311a-c7ed-42b1-8388-3aa860716c5c-podres\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.291944 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.291910    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk8ps\" (UniqueName: \"kubernetes.io/projected/2444311a-c7ed-42b1-8388-3aa860716c5c-kube-api-access-xk8ps\") pod \"perf-node-gather-daemonset-2d8hn\" (UID: \"2444311a-c7ed-42b1-8388-3aa860716c5c\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.353312 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.353270    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.557471 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.557438    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"]
Jul 29 17:35:32.559682 ip-10-0-134-156 kubenswrapper[2570]: W0729 17:35:32.559646    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2444311a_c7ed_42b1_8388_3aa860716c5c.slice/crio-4c88b0c1ec896455863ca37d8278f9556277641d27e54ed23233415c047c7554 WatchSource:0}: Error finding container 4c88b0c1ec896455863ca37d8278f9556277641d27e54ed23233415c047c7554: Status 404 returned error can't find the container with id 4c88b0c1ec896455863ca37d8278f9556277641d27e54ed23233415c047c7554
Jul 29 17:35:32.763529 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.763488    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn" event={"ID":"2444311a-c7ed-42b1-8388-3aa860716c5c","Type":"ContainerStarted","Data":"c921c95e0f56ec85e0a11c700515e267bdc3a10dff155fe80b992dc408abf3a1"}
Jul 29 17:35:32.763734 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.763559    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn" event={"ID":"2444311a-c7ed-42b1-8388-3aa860716c5c","Type":"ContainerStarted","Data":"4c88b0c1ec896455863ca37d8278f9556277641d27e54ed23233415c047c7554"}
Jul 29 17:35:32.764024 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.763999    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:32.781715 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:32.781655    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn" podStartSLOduration=0.781633103 podStartE2EDuration="781.633103ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 17:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:35:32.779014813 +0000 UTC m=+2661.268534470" watchObservedRunningTime="2026-07-29 17:35:32.781633103 +0000 UTC m=+2661.271152761"
Jul 29 17:35:33.321031 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:33.320998    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-95x46_b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85/dns/0.log"
Jul 29 17:35:33.343625 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:33.343599    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-95x46_b6e4b6d0-93b5-4b86-9249-9ba2f27c8c85/kube-rbac-proxy/0.log"
Jul 29 17:35:33.419319 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:33.419289    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dq655_8ed06670-f24d-4521-815a-0d27741edac3/dns-node-resolver/0.log"
Jul 29 17:35:34.035186 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:34.035139    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6qp9p_e22624ca-991d-4e4d-8db7-02119caf746d/node-ca/0.log"
Jul 29 17:35:35.132941 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:35.132910    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5df7c8cd4c-mgz6p_fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6/router/0.log"
Jul 29 17:35:35.784452 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:35.784425    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zl259_240795ba-8cfb-4ab3-b9ac-6d453268441f/serve-healthcheck-canary/0.log"
Jul 29 17:35:36.276486 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:36.276448    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-zx4jr_af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e/insights-operator/0.log"
Jul 29 17:35:36.278418 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:36.278393    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-zx4jr_af34f6cb-d6d1-4cd1-b9b3-069f411fdd3e/insights-operator/1.log"
Jul 29 17:35:36.418098 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:36.418070    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fwrgv_84e4fe40-6165-40b3-9f20-3d3003e57759/kube-rbac-proxy/0.log"
Jul 29 17:35:36.441348 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:36.441318    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fwrgv_84e4fe40-6165-40b3-9f20-3d3003e57759/exporter/0.log"
Jul 29 17:35:36.464717 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:36.464686    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fwrgv_84e4fe40-6165-40b3-9f20-3d3003e57759/extractor/0.log"
Jul 29 17:35:38.781912 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:38.781884    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-2d8hn"
Jul 29 17:35:40.249559 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:40.249529    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-cd5d4f59d-4f2hx_26130c76-a292-4866-a4f1-2d254fc60d0b/manager/0.log"
Jul 29 17:35:40.321726 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:40.321681    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-d6cdfdb88-kq6gb_7860416e-8673-4117-90b2-0aff71370ac8/manager/0.log"
Jul 29 17:35:40.350326 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:40.350298    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-k9vr4_31aa38fa-7343-498c-8e17-9a7fd0c07b7b/server/0.log"
Jul 29 17:35:40.753175 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:40.753123    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-kl5hm_3e3914fc-3283-494d-9546-4ef0cffb78aa/manager/0.log"
Jul 29 17:35:40.776817 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:40.776786    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-6kv5c_71d092b8-2461-4a6c-8f00-54ac4f4b94c6/s3-init/0.log"
Jul 29 17:35:40.814217 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:40.813965    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-8445dbf85d-s6kvn_db91b8cd-9b2d-4caa-867b-c32a4346a28b/seaweedfs/0.log"
Jul 29 17:35:43.324990 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:43.324962    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5df7c8cd4c-mgz6p_fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6/router/0.log"
Jul 29 17:35:44.415528 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:44.415494    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5df7c8cd4c-mgz6p_fb3c57cf-294f-4d44-bb5d-8edfb8a0b7d6/router/0.log"
Jul 29 17:35:46.442988 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:46.442953    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-7httm_2c5076ed-8a8a-4866-801a-5b0037ec4d36/kube-storage-version-migrator-operator/1.log"
Jul 29 17:35:46.445471 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:46.445442    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-7httm_2c5076ed-8a8a-4866-801a-5b0037ec4d36/kube-storage-version-migrator-operator/0.log"
Jul 29 17:35:47.616594 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:47.616555    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-87796_0e13b680-aa89-4a92-8ff2-1eb3e0a0309b/kube-multus/0.log"
Jul 29 17:35:48.025693 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:48.025662    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lsxjp_e356ec32-854a-4b20-985b-e93890a39aa3/kube-multus-additional-cni-plugins/0.log"
Jul 29 17:35:48.047582 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:48.047553    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lsxjp_e356ec32-854a-4b20-985b-e93890a39aa3/egress-router-binary-copy/0.log"
Jul 29 17:35:48.068376 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:48.068348    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lsxjp_e356ec32-854a-4b20-985b-e93890a39aa3/cni-plugins/0.log"
Jul 29 17:35:48.097902 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:48.097868    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lsxjp_e356ec32-854a-4b20-985b-e93890a39aa3/bond-cni-plugin/0.log"
Jul 29 17:35:48.120495 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:48.120466    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lsxjp_e356ec32-854a-4b20-985b-e93890a39aa3/routeoverride-cni/0.log"
Jul 29 17:35:48.144155 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:48.144127    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lsxjp_e356ec32-854a-4b20-985b-e93890a39aa3/whereabouts-cni-bincopy/0.log"
Jul 29 17:35:48.168393 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:48.168358    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lsxjp_e356ec32-854a-4b20-985b-e93890a39aa3/whereabouts-cni/0.log"
Jul 29 17:35:48.233992 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:48.233957    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pmqs8_438c3501-4812-489f-ae51-a25e01b9aacc/network-metrics-daemon/0.log"
Jul 29 17:35:48.252874 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:48.252842    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pmqs8_438c3501-4812-489f-ae51-a25e01b9aacc/kube-rbac-proxy/0.log"
Jul 29 17:35:49.448502 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:49.448469    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qfpbq_9243e853-a0da-492f-b653-b83b2d974747/ovn-controller/0.log"
Jul 29 17:35:49.467466 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:49.467437    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qfpbq_9243e853-a0da-492f-b653-b83b2d974747/ovn-acl-logging/0.log"
Jul 29 17:35:49.494866 ip-10-0-134-156 kubenswrapper[2570]: I0729 17:35:49.494831    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qfpbq_9243e853-a0da-492f-b653-b83b2d974747/kube-rbac-proxy-node/0.log"